Optics for video cameras on a surgical visualization system

ABSTRACT

A surgical device includes a plurality of cameras integrated therein. The view of each of the plurality of cameras can be integrated together to provide a composite image. A surgical tool that includes an integrated camera may be used in conjunction with the surgical device. The image produced by the camera integrated with the surgical tool may be associated with the composite image generated by the plurality of cameras integrated in the surgical device. The position and orientation of the cameras and/or the surgical tool can be tracked, and the surgical tool can be rendered as transparent on the composite image. A surgical device may be powered by a hydraulic system, thereby reducing electromagnetic interference with tracking devices.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority under 35 U.S.C. §119(e)to U.S. Provisional Application No. 61/665,243, filed Jun. 27, 2012,U.S. Provisional Application No. 61/670,550, filed Jul. 11, 2012, U.S.Provisional Application No. 61/703,727, filed Sep. 20, 2012, and U.S.Provisional Application No. 61/753,398, filed Jan. 16, 2013. Each ofthese applications is hereby incorporated by reference in its entirety.

BACKGROUND

1. Field

Embodiments of the present disclosure relate to surgical devices andvisualization systems for use during surgery.

2. Description of Related Art

Some surgical operations involve the use of large incisions. These opensurgical procedures provide ready access for surgical instruments andthe hand or hands of the surgeon, allowing the user to visually observeand work in the surgical site, either directly or through an operatingmicroscope or with the aide of loupes. Open surgery is associated withsignificant drawbacks, however, as the relatively large incisions resultin pain, scarring, and the risk of infection as well as extendedrecovery time. To reduce these deleterious effects, techniques have beendeveloped to provide for minimally invasive surgery. Minimally invasivesurgical techniques, such as endoscopy, laparoscopy, arthroscopy,pharyngo-laryngoscopy, as well as small incision procedures utilizing anoperating microscope for visualization, utilize a significantly smallerincision than typical open surgical procedures. Specialized tools maythen be used to access the surgical site through the small incision.However, because of the small access opening, the surgeon's view andworkspace of the surgical site is limited. In some cases, visualizationdevices such as endoscopes, laparoscopes, and the like can be insertedpercutaneously through the incision to allow the user to view thesurgical site. Alternatively operating microscopes may be used to viewthe surgical site through a small incision held open by one or a numberof surgical retractors.

The visual information available to a user through laparoscopic,endoscopic, or operating microscope contain trade-offs in approach.Accordingly, there is a need for improved visualization systems, for usein minimally invasive surgery.

SUMMARY OF THE INVENTION

The systems, methods and devices of the disclosure each have severalinnovative aspects, no single one of which is solely responsible for thedesirable attributes disclosed herein.

In accordance with one aspect, a medical apparatus comprises a surgicalretractor configured to hold open an incision and thereby provide apathway for access of surgical tools to a surgical site; and a pluralityof cameras disposed on the surgical retractor, the cameras inwardlyfacing toward the pathway. In some embodiments, the surgical retractorcan comprise a plurality of retractor blades and the cameras aredisposed on the retractor blades. In some embodiments, the surgicalretractor comprises a tube and the cameras can be disposed on an insidesurface of the tube. In some embodiments, the surgical retractor cancomprise: a proximal camera at a proximal location; and a distal cameraat a distal location; wherein the distal location is configured to bedisposed closer to the surgical site than the proximal location. In someembodiments, the medical apparatus can further comprise a plurality ofproximal cameras at the proximal location and a plurality of distalcameras at the distal location. In some embodiments, the plurality ofproximal cameras can comprise at least 3 cameras. In some embodiments,the plurality of distal cameras can comprise at least 3 cameras. In someembodiments, the surgical retractor can be configured for use in spinesurgery. In some embodiments, the surgical retractor can be configuredfor use in head or neck surgery. In some embodiments, the surgicalretractor can be configured for use in neurosurgery. In someembodiments, the plurality of cameras can comprise at least 8 cameras.

In accordance with another aspect, a method comprises: inserting aretractor into an opening in a body; holding open edges of the openingwith the retractor, thereby providing a pathway for access of surgicaltools to a surgical site; and inserting a surgical tool at leastpartially through pathway and to the surgical site, wherein theretractor comprises a plurality of cameras disposed on the surgicalretractor, the cameras inwardly facing toward the pathway. In someembodiments, the surgical tool can comprise a scalpel, a rongeur, akerrison, a laser, or a drill. In some embodiments, the surgical site isa portion of a spine of the body. In some embodiments, the surgical siteis in a head or in a neck of the body. In some embodiments, the surgicalsite is in a brain of the body. In some embodiments, the opening is amouth of the body. In some embodiments, the retractor comprises aplurality of retractor blades and the cameras are disposed on theretractor blades. In some embodiments, the retractor comprises a tubeand the cameras are disposed on an inside surface of the tube. In someembodiments, the method further comprises removing bone at the surgicalsite with the surgical tool.

In accordance with another aspect, a surgical visualization systemcomprises: a retractor; a plurality of cameras disposed on theretractor, said cameras producing respective images; and an imageprocessing module configured to display said respective images, whereinsaid surgical visualization system is configured to track the locationof said cameras. In some embodiments, said cameras are associated withtracking devices to track the relative location of the differentcameras. In some embodiments, said tracking devices compriseelectromagnetic (EM) tracking devices. In some embodiments, the camerasare disposed on movable blades of the retractor. In some embodiments,the surgical visualization system tracks the location of the pluralityof cameras by tracking the position of the retractor blades. In someembodiments, the retractor blades comprise a plurality of segmentsconnected by at least one hinge. In some embodiments, the retractorblades are malleable. In some embodiments, the cameras are removablycoupled to the retractor. In some embodiments, the cameras areassociated with tracking devices to track the relative locations of thedifferent cameras, and the tracking devices are removably coupled to theretractor.

In accordance with another aspect, a method comprises: inserting aretractor into an opening in a body, the retractor comprising aplurality of cameras disposed thereon; and electronically tracking thelocations of said cameras. In some embodiments, the method furthercomprises obtaining respective images from each of the plurality ofcameras, and processing said respective images for simultaneous viewing.In some embodiments, said processing comprises using the trackedlocations of said cameras. In some embodiments, said electronicallytracking comprises electromagnetic (EM) tracking. In some embodiments,said electronically tracking comprises tracking the position of a firstcamera relative to the position of a second camera. In some embodiments,said cameras are disposed on blades of the retractor, and saidelectronically tracking comprises tracking position of the blades. Insome embodiments, said blades are movable with respect to one another.In some embodiments, said blades are malleable or articulated. In someembodiments, said electronically tracking comprises tracking the degreeto which said blades are bent or articulated.

In accordance with another aspect, a medical apparatus comprises: asurgical retractor configured to provide access to a surgical site; aplurality of cameras disposed on the retractor; a plurality of trackingdevices configured to track the locations of at least some of theplurality of cameras. In some embodiments, the plurality of camerascomprises at least a first camera and a second camera each have trackingdevices associated therewith to track the relative positions of thefirst and second camera with respect to each other. In some embodiments,the retractor comprises at least a first retractor blade and a secondretractor blade each having at least one camera thereon, and each ofsaid first and second retractor blades have tracking devices thereon totrack the relative positions of the first and second retractor bladeswith respect to each other. In some embodiments, said tracking devicescomprise electromagnetic (EM) tracking devices.

In accordance with another aspect, a medical apparatus comprises: asurgical retractor configured to hold open an incision and therebyprovide a pathway for access of surgical tools to a surgical site; and aplurality of cameras disposed on the surgical retractor, said camerasinwardly facing toward said pathway. In some embodiments, said surgicalretractor comprises a plurality of retractor blades and said cameras aredisposed on said retractor blades. In some embodiments, said surgicalretractor comprises a tube and said cameras are disposed on an insidesurface of said tube. In some embodiments, said surgical retractorcomprises at least one proximal camera and at least one distal camera.In some embodiments, the cameras are fastened to the surgical retractorusing a clip-on fastener. In some embodiments, said clip-on fastenercomprises a clip, a snap, a screw, a bolt, a nut, magnet. In someembodiments, the medical apparatus further comprises an electrical busand electrical connector configured to connect said cameras to saidelectrical bus. In some embodiments, said electrical bus is fastened tosaid surgical retractor using a clip-on fastener. In some embodiments,the cameras are integrated into the surgical retractor. In someembodiments, said cameras produce respective images, and the apparatusfurther comprises an image processing module. In some embodiments, theimage processing module is configured to display at least a portion ofsaid respective images as stitched together. In some embodiments, theimage processing module is configured to display at least a portion ofsaid respective images as tiled. In some embodiments, the imageprocessing module is configured to display said respective tiled imagesas arranged with a first central image and a plurality of surroundingimages, said first central image from a different camera than saidplurality of surrounding images. In some embodiments, said first centralimage and a plurality of surrounding images correspond to a firstcentral view and a plurality peripheral views. In some embodiments, saidfirst central image has larger zoom magnification than said pluralityperipheral views. In some embodiments, said first central imagecomprises at least a portion of said respective images stitchedtogether. In some embodiments, said first central image comprises astereo image. In some embodiments, said plurality of surrounding imagescomprises at least a portion of said respective images tiled together.In some embodiments, said plurality of surrounding images comprises atleast a portion of said respective images stitched together. In someembodiments, the image processing module is configured to display saidrespective images as arranged with a main view and a plurality of imagessuperimposed on said main view, said main view from a different camerathan said plurality of superimposed images. In some embodiments, saidfirst main view covers a larger field-of-view than any of said othersuperimposed images. In some embodiments, said first main view comprisesa first background view. In some embodiments, the apparatus furthercomprises a picture-in-picture in said main view, saidpicture-in-picture having a larger zoom magnification than said mainview. In some embodiments, said picture-in-picture comprises pluralityof said images stitched together. In some embodiments, the imageprocessing module is further configured to display a first central imagedisposed centrally with respect to said main view and said plurality ofsuperimposed images. In some embodiments, said first central images hasa larger zoom magnification than said main view. In some embodiments,said main view comprises at least a portion of at least a plurality ofsaid respective images stitched together. In some embodiments, said mainview comprises at least a portion of at least a plurality of saidrespective images tiled together. In some embodiments, said plurality ofsuperimposed images comprises at least a portion of at least a pluralityof said respective images tiled together. In some embodiments, saidplurality of superimposed images comprises at least a portion of saidrespective images stitched together. In some embodiments, saidrespective images are arranged with a first main view and an imagesuperimposed on said main view, said main view from a different camerathan said superimposed image, said first main view comprising a largerfield-of-view than said superimposed image. In some embodiments, saidfirst main view comprises covers a larger field of view than any of saidother superimposed images. In some embodiments, said first main viewcomprises a first background view. In some embodiments, saidsuperimposed image comprises a picture-in-picture image. In someembodiments, said image processing module is configured to receiveselection of views from a user via an interface. In some embodiments,said image processing module is configured such that a user can select aplurality of views and the image processor at least partially locatesthe views based on the location of said cameras with respect to eachother. In some embodiments, said image processing module is configuredsuch that a user can specify a mode and the image processor provides acollection of views associated with that mode without the userindependently specifying the images or sensors.

In accordance with another aspect, a method comprises: inserting aretractor into an opening in a body; holding open edges of the openingwith the retractor, thereby providing a pathway for access of surgicaltools to a surgical site; and inserting a surgical tool at leastpartially through pathway and to the surgical site, wherein saidretractor comprises a plurality of cameras disposed on the surgicalretractor, said cameras inwardly facing toward said pathway. In someembodiments, the surgical tool comprises a scalpel, a rongeur, akerrison, a laser, or a drill. In some embodiments, the plurality ofcameras comprises at least 8 cameras. In some embodiments, the surgicalsite is an area of the spine of the body. In some embodiments, thesurgical site is an area of a head or neck of the body. In someembodiments, the retractor comprises a plurality of retractor blades andsaid cameras are disposed on said retractor blades. In some embodiments,the retractor comprises a tube and said cameras are disposed on aninside surface of the tube. In some embodiments, the surgical retractorcomprises at least one proximal camera and at least one distal camera.In some embodiments, the method further comprises obtaining respectiveimages produced by the cameras. In some embodiments, the method furthercomprises displaying said respective images simultaneously. In someembodiments, the respective images are displayed as stitched together.In some embodiments, the respective images are displayed as tiled.

In accordance with another aspect, a medical apparatus comprises: asurgical retractor configured to hold open an incision formed in a bodyand thereby provide a pathway for access for surgical tools to asurgical site in said body, said retractor including proximal and distallocations, said distal location configured to be disposed further withinsaid body than said proximal location; and a plurality of camerasdisposed on the surgical retractor, including at least one proximalcamera at said proximal location and at least one distal camera at saiddistal location. In some embodiments, said surgical retractor comprisesa plurality of retractor blades and said cameras are disposed on atleast one proximal and at least one distal location of said retractorblades. In some embodiments, said surgical retractor comprises a tubeand said cameras are disposed on at least one proximal and at least onedistal location on inside surface of said tube. In some embodiments,said cameras are fastened to the surgical retractor using a clip-onfastener. In some embodiments, said clip-on fastener comprises a clip, asnap, a screw, a bolt, a nut, or a magnet. In some embodiments, themedical apparatus further comprises an electrical bus and electricalconnector configured to connect said cameras to said electrical bus. Insome embodiments, said cameras are integrated into the surgicalretractor. In some embodiments, the proximal camera and the distalcamera produce respective images, and the apparatus further comprises animage processing module configured to display respective images forsimultaneous viewing. In some embodiments, the proximal camera isoriented along a first optical axis, and the distal camera is orientedalong a second optical axis, and the first and second optical axes aresubstantially parallel. In some embodiments, the proximal camera isoriented along a first optical axis, and wherein the distal camera isoriented along a second optical axis, and wherein the first and secondoptical axes intersect at a first point. In some embodiments, the firstpoint is within the pathway for access for surgical tools to thesurgical site. In some embodiments, the proximal camera is orientedalong a first optical axis, wherein the first optical axis issubstantially orthogonal to a plane of a surface of the retractor at theproximal location. In some embodiments, the distal camera is orientedalong a second optical axis, wherein the second optical axis issubstantially orthogonal to a plane of a surface of the retractor at thedistal location. In some embodiments, the proximal camera provides afirst field-of-view and the distal camera provides a second field ofview, the second field of view being smaller than the firstfield-of-view. In some embodiments, the first and second fields of viewat least partially overlap. In some embodiments, the first field of viewis between about 80 and 100 degrees. In some embodiments, the secondfield of view is between about 50 and 70 degrees. In some embodiments,the retractor is configured such that when holding open the incisionformed in the body and thereby providing a pathway for access forsurgical tools to the surgical site in said body, the first and secondfields of view each encompass at least a portion of the surgical site.In some embodiments, said cameras produce respective images, and whereinthe apparatus further comprises an image processing module configured todisplay respective images for simultaneous viewing. In some embodiments,the image processing module is configured to display at least a portionof said respective images as stitched together. In some embodiments, theimage processing module is configured to display at least a portion ofsaid respective images as tiled.

In accordance with another aspect, a surgical visualization systemcomprises: a surgical retractor having proximal and distal locations,said distal location configured to be disposed further within a bodythan said proximal location; a proximal camera disposed at said proximallocation; and a distal camera disposed at said distal location. In someembodiments, said surgical retractor comprises a plurality of retractorblades and said cameras are disposed on at least one proximal and atleast one distal location of said retractor blades. In some embodiments,said surgical retractor comprises a tube and said cameras are disposedon at least one proximal and at least one distal location on insidesurface of said tube.

In accordance with another aspect, a method comprises: inserting aretractor into an opening in a body; holding open edges of the openingwith the retractor, thereby providing a pathway for access of surgicaltools to a surgical site; and inserting a surgical tool at leastpartially through pathway, wherein said retractor comprises a proximalcamera and a distal camera, said cameras inwardly facing toward saidpathway. In some embodiments, the surgical tool comprises a scalpel, arongeur, a kerrison, a laser, or a drill. In some embodiments, theretractor comprises a plurality of proximal cameras and a plurality ofdistal cameras. In some embodiments, the surgical site is an area of thespine of the body. In some embodiments, the surgical site is an area ofa head or neck of the body.

In accordance with another aspect, a surgical visualization systemcomprises: a retractor having a plurality of cameras disposed thereon; asurgical tool having at least one camera disposed thereon; and an imageprocessing module configured to display respective images from saidplurality of cameras on said retractor and said camera on said surgicaltool for simultaneous viewing. In some embodiments, the surgical tool ismovable with respect to the retractor. In some embodiments, at least oneof the plurality of cameras substantially faces another one of theplurality of cameras. In some embodiments, the image processing moduleis configured to display said respective images for simultaneous viewingas a composite first image. In some embodiments, the image processingmodule is configured to integrate a second image obtained from the atleast one camera on the surgical tool with the composite first image. Insome embodiments, the composite first image is produced by tiling orstitching the respective images from the plurality of cameras on theretractor. In some embodiments, the image processing module isconfigured to display the second image as a picture-in-picture over thecomposite first image. In some embodiments, the image processing moduleis configured to stitch the second image with the composite first imageto produce a composite third image. In some embodiments, the pluralityof cameras are attached to a surface of blades of the retractor. In someembodiments, the plurality of cameras are integrated within blades ofthe retractor. In some embodiments, the plurality of cameras comprisesat least 8 cameras. In some embodiments, the surgical tool comprises ascalpel, a rongeur, a kerrison, a laser, or a drill.

In accordance with another aspect, a surgical visualization systemcomprises: a retractor having a plurality of cameras disposed thereon; acamera configured to be associated with a surgical tool; and an imageprocessing module configured to display respective images from saidplurality of cameras on said retractor and images from said cameraassociated with said surgical tool for simultaneous viewing. In someembodiments, at least one of the plurality of cameras disposed on theretractor substantially faces another one of the plurality of camerasdisposed on the retractor. In some embodiments, the image processingmodule is configured to display said respective images for simultaneousviewing as a composite first image. In some embodiments, the imageprocessing module is configured to integrate a second image obtainedfrom the camera configured to be associated with the surgical tool withthe composite first image.

In accordance with another aspect, a method comprises: receiving from aplurality of cameras disposed on a retractor a first plurality of imagedata; receiving from a camera disposed on a surgical tool a secondplurality of image data; processing the first plurality of image data toproduce a first image; and processing the second plurality of image datato produce a second image. In some embodiments, producing the firstimage comprises stitching or tiling separate images obtained from theplurality of cameras. In some embodiments, the method further comprisesintegrating the second image with the first image. In some embodiments,integrating comprises disposing the second image as a picture-in-pictureover the first image. In some embodiments, integrating comprisesstitching the second image with the first image to produce a compositethird image.

In accordance with another aspect, a surgical visualization kitcomprises: a plurality of cameras configured to be disposed on aretractor, said cameras configured to produce respective images; animage processing module configured display said respective images; and acamera configured to be disposed on a surgical tool. In someembodiments, the image processing module is configured to display saidrespective images for simultaneous viewing as a composite first image.In some embodiments, displaying said composite first image is producedby stitching or tiling the respective images produced by the pluralityof cameras. In some embodiments, the image processing module is furtherconfigured to display a second image obtained from the camera configuredto be disposed on the surgical tool. In some embodiments, the imageprocessing module is configured to display the second image as apicture-in-picture over the first image. In some embodiments, the imageprocessing module is configured to stitch the second image with thefirst image to produce a composite third image.

In accordance with another aspect, a method comprises: inserting aretractor into an opening in a body, the retractor comprising aplurality of cameras disposed thereon; inserting a surgical tool atleast partially into a working space of the retractor, the surgical toolcomprising at least one camera disposed thereon. In some embodiments, atleast some of the plurality of cameras are disposed on a blade of theretractor and substantially face the working space. In some embodiments,the plurality of cameras comprises at least 8 cameras.

In accordance with another aspect, a medical apparatus comprises: asurgical retractor configured to hold open an opening in a body andthereby provide a pathway for access of surgical tools to a surgicalsite, wherein the retractor comprises a rotatable platform; and aplurality of cameras disposed on the rotatable platform. In someembodiments, said retractor comprises retractor blades configured tohold open the opening. In some embodiments, said rotatable platform ismovable with respect to the retractor blades. In some embodiments, saidretractor blades move with rotation of said rotatable platform. In someembodiments, said retractor comprises a proximal end and a distal end,said distal end configured to be disposed further within said body, andwherein said rotatable platform is arranged proximal to the retractorblades. In some embodiments, the apparatus further comprises a secondplurality of cameras disposed on the retractor blades. In someembodiments, said cameras are configured to produce respective images,the apparatus further comprising an image processing module configuredto display respective images simultaneously. In some embodiments,rotation of said rotatable platform produces rotation of thesimultaneous display of the respective images. In some embodiments, theimage processing module is configured to display at least a portion ofsaid respective images as stitched together. In some embodiments, theimage processing module is configured to display at least a portion ofsaid respective images as tiled. In some embodiments, the plurality ofcameras comprises a first camera and a second camera, wherein the firstcamera has a first field of view, and the second camera has a secondfield of view. In some embodiments, the retractor is configured such thefirst field of view encompasses the surgical site. In some embodiments,the retractor is configured such the first and second fields of vieweach encompass the surgical site. In some embodiments, the retractor isconfigured such the first and second fields of view at least partiallyoverlap. In some embodiments, at least one of the plurality of camerassubstantially faces the pathway for access of surgical tools.

In accordance with another aspect, a surgical visualization systemcomprises: a surgical retractor having proximal and distal locations,said distal location configured to be disposed further within a bodythan said proximal location, said retractor comprising a rotatableplatform; and a plurality of cameras disposed on the rotatable platform.In some embodiments, said retractor comprises retractor bladesconfigured to hold open the opening, wherein the retractor blades aredistal to the rotatable platform. In some embodiments, said rotatableplatform is rotatable with respect to the retractor blades. In someembodiments, said retractor comprises a tube configured to hold open theopening, wherein the tube is distal to the rotatable platform. In someembodiments, said rotatable platform is rotatable with respect to thetube. In some embodiments, the retractor defines a pathway for access ofsurgical tools to a surgical site, and wherein at least one of thecameras substantially faces the surgical site.

In accordance with another aspect, a method comprises: inserting aretractor at least partially into an opening in a body, the retractorcomprising a rotatable platform having a plurality of cameras thereon;holding open edges of the opening with the retractor, thereby providinga pathway for access of surgical tools to a surgical site; and rotatingthe rotatable platform, thereby altering the orientation of theplurality of cameras with respect to the opening. In some embodiments,the rotatable platform is disposed outside the opening of the body. Insome embodiments, the surgical site is an area of the spine of the body.In some embodiments, the surgical site is an area of a head or neck ofthe body. In some embodiments, the opening is a mouth of the body. Insome embodiments, the plurality of cameras comprises at least 8 cameras.In some embodiments, the retractor comprises a plurality of retractorblades, and wherein upon insertion of the retractor at least partiallyinto the opening, said retractor blades are closer to said surgical sitethan said rotatable platform. In some embodiments, the retractorcomprises a tube, and upon insertion of the retractor at least partiallyinto the opening, said tube is closer to said surgical site than saidrotatable platform.

In accordance with another aspect, a medical apparatus comprises: asurgical device; at least one camera disposed on the surgical device;and a hydraulic system configured to deliver fluid pulses to the atleast one camera. In some embodiments, the surgical device is aretractor. In some embodiments, the fluid comprises water. In someembodiments, the fluid comprises pharmaceuticals, fluorescent dyes, orsaline. In some embodiments, the fluid pulses are configured to removeobstructions from the at least one camera. In some embodiments, theapparatus further comprises a plurality of cameras, wherein thehydraulic system is configured to deliver fluid pulses to each of theplurality of cameras. In some embodiments, the hydraulic systemcomprises a plurality of microfluidic channels coupled to a fluidsource. In some embodiments, the microfluidic channels comprise a flexcable configured to be positioned over an electronic cable. In someembodiments, the microfluidic channels are disposable. In someembodiments, the distal end of the flex cable comprises an outer housingsecured over the camera. In some embodiments, the shape of the outerhousing is configured to direct fluid from the flex cable over a surfaceof the camera. In some embodiments, hydraulic system comprises adisposable diaphragm pump. In some embodiments, the hydraulic systemcomprises at least one of: a rolling edge diaphragm, Bourdon tube, or abellow. In some embodiments, the at least one camera comprises a lensincluding a stop behind a plano window, and wherein the hydraulic systemis configured to deliver a fluid pulse over the plano window. In someembodiments, the hydraulic system is further configured to deliverpulses of pressurized air to the camera. In some embodiments, the pulsesof pressurized air are configured to dry the camera following the fluidpulses. In some embodiments, the hydraulic system is controlled by aproportional foot pedal. In some embodiments, the hydraulic system isfurther configured to provide egress of gases and/or liquids.

In accordance with another aspect, a surgical visualization systemcomprises: a surgical retractor having a plurality of cameras disposedthereon; a surgical tool having at least one camera disposed thereon;and an image processing module configured to receive signals from saidcameras on said retractor and said surgical tool for display ofrespective images from said cameras, wherein said image processingmodule is configured to track the locations of said plurality of camerasand of said surgical tool. In some embodiments, said surgical tool ismovable with respect to said surgical retractor. In some embodiments,said surgical tool is associated with a tracking device to track thelocation of the surgical tool. In some embodiments, said tracking devicecomprises an EM tracking device. In some embodiments, said camerasdisposed on the surgical retractor are associated with tracking devicesto track the location of the cameras. In some embodiments, said trackingdevices comprise EM tracking devices. In some embodiments, the imageprocessing module is configured to track the location of said surgicaltool with optical tracking. In some embodiments, the system furthercomprises an overhead camera, wherein the surgical tool includesidentifying markers visible to the overhead camera, and wherein theimage processing module is configured to track the location of thesurgical tool by tracking the identifying markers on the surgical tool.In some embodiments, the image processing module is configured todisplay said respective images simultaneously. In some embodiments, theimage processing module is configured to adjust the display of saidrespective images depending upon the tracked location of said surgicaltool. In some embodiments, the image processing module is configured todisplay respective images obtained from the cameras disposed on saidretractor for simultaneous viewing as a composite first image. In someembodiments, the image processing module is configured to display animage obtained from the camera disposed on said surgical toolsimultaneously with the composite first image. In some embodiments, theimage processing module is configured to display the image obtained fromthe camera disposed on said surgical tool as a picture-in-picture overthe composite first image. In some embodiments, the image processingmodule is configured to stitch the image obtained from the cameradisposed on said surgical tool with the first composite image to producea second composite image. In some embodiments, the plurality of camerasare attached to a surface of blades of the retractor. In someembodiments, the plurality of cameras face inwardly towards a pathwaydefined by the blades of the retractor. In some embodiments, the bladesof the retractor are malleable or articulated. In some embodiments, theblades of the retractor are movable with respect to one another.

In accordance with another aspect, a method comprises: inserting aretractor into an opening in a body, the retractor comprising aplurality of cameras disposed thereon, wherein the retractor defines apathway for access of surgical tools to a surgical site within the body;inserting a surgical tool into the pathway, the surgical tool;electronically tracking the locations of said cameras and the locationof said surgical tool. In some embodiments, the surgical tool comprisesa camera disposed thereon. In some embodiments, the method furthercomprises obtaining respective images from each of the cameras, andprocessing said respective images for simultaneous viewing. In someembodiments, said processing comprises using the tracked locations ofsaid cameras. In some embodiments, said electronically trackingcomprises EM tracking. In some embodiments, said plurality of camerasare disposed on blades of the retractor, and wherein said electronicallytracking comprises tracking position of the blades. In some embodiments,said blades are malleable or articulated. In some embodiments, saidelectronically tracking comprises tracking the degree to which saidblades are bent or articulated.

In accordance with another aspect, a medical apparatus can comprise asurgical device, a hydraulic system providing hydraulic power to saidsurgical device, the hydraulic system comprising, a hydraulic fluidsource, and a cassette assembly having a plurality of external fluidports, one or more hydraulic pressure chambers, and a plurality ofvalves positioned on one or more fluid paths fluidly connecting theexternal fluid ports to the one or more hydraulic pressure chambers, thehydraulic fluid source being in fluid communication with the one or morehydraulic pressure chambers via one or more of the external fluid ports,and an electromagnetic tracking device. In some embodiments, theelectromagnetic tracking device can be configured to track said surgicaldevice. In some embodiments, electromagnetic tracking device can beconfigured to track camera modules on a retractor. In some embodiments,one or more of the plurality of valves can be a diaphragm valve. In someembodiments, wherein one or more of the plurality of valves can be aproportional valve. In some embodiments, one or more of the plurality ofvalves can be an elastomeric valve. In some embodiments, the cassetteassembly can comprise disposable components. In some embodiments, theentire cassette assembly can be disposable. In some embodiments, thehydraulic system can further comprise a hydraulic turbine operablyconnected to the surgical device to actuate the surgical device. In someembodiments, the apparatus can further comprise one or more washingnozzles in fluid communication with one or more of the hydraulicpressure chambers or the hydraulic fluid source, the one or more washingnozzles configured to direct hydraulic fluid toward one or more lightsources.

In accordance with another aspect, a medical apparatus can comprise asurgical device, a hydraulic system providing hydraulic power to saidsurgical device, the hydraulic system comprising, a hydraulic fluidsource, and a cassette assembly having a plurality of external fluidports, one or more hydraulic pressure chambers, and a plurality ofvalves positioned on one or more fluid paths fluidly connecting theexternal fluid ports to the one or more hydraulic pressure chambers, thehydraulic fluid source being in fluid communication with the one or morehydraulic pressure chambers via one or more of the external fluid ports,and one or more cameras for providing a view of an area in the body. Insome embodiments, In some embodiments, the apparatus can furthercomprise an electromagnetic tracking device. In some embodiments, saidelectromagnetic tracking device can be configured to track cameramodules on a retractor. In some embodiments, said camera can be on saidsurgical device. In some embodiments, said camera can be on a retractorin said area in the body. In some embodiments, one or more of theplurality of valves can be a diaphragm valve. In some embodiments, oneor more of the plurality of valves can be a proportional valve. In someembodiments, one or more of the plurality of valves can be anelastomeric valve. In some embodiments, the cassette assembly cancomprise disposable components. In some embodiments, the entire cassetteassembly can be disposable. In some embodiments, the hydraulic systemcan further comprise a hydraulic turbine operably connected to thesurgical device to actuate the surgical device.

In accordance with another aspect, a method of tracking a surgicaldevice, the method can comprise providing a surgical device, operablyconnecting the surgical device to a hydraulic system, the hydraulicsystem can comprise: a hydraulic fluid source, and a cassette assemblyhaving a plurality of external fluid ports, one or more hydraulicpressure chambers, and a plurality of valves positioned on one or morefluid paths fluidly connecting the external fluid ports to the one ormore hydraulic pressure chambers, the hydraulic fluid source being influid communication with and providing hydraulic fluid to the one ormore hydraulic pressure chambers via one or more of the external fluidports; pressurizing the hydraulic fluid; tracking the surgical deviceusing one or more of a camera and an electromagnetic tracking device. Insome embodiments, the method of tracking a surgical device can furthercomprise operably connecting a hydraulic turbine of the surgical deviceto the hydraulic system.

In accordance with another aspect, a medical apparatus comprises: aretractor stage comprising a ring defining an aperture, the ringsubstantially aligned with a first plane substantially orthogonal to afirst axis; a plurality of blades coupled to said stage and positionedwithin said aperture, each of the blades extending away from the firstplane, wherein each of the blades is configured to be: rotationallymoved with respect to said ring; radially moved inward and outward withrespect to said ring; and tilted with respect to said first axis. Insome embodiments, each of the blades is configured to be tilted byflexing. In some embodiments, each of the blades is be jointed, and eachof the blades is configured to be titled by bending at a joint. In someembodiments, each of the blades is coupled to said stage by a stemextending between a proximal end of the blade and the stage. In someembodiments, the stem is coupled to the stage by a clamp. In someembodiments, the clamp is configured to be moved rotationally around thering, thereby rotationally moving the stem and retractor. In someembodiments, the stem is configured to be slidably moved through theclamp, thereby moving the stem and the retractor radially inward oroutward with respect to said ring. In some embodiments, the apparatusfurther comprises a plurality of cameras disposed on the retractorblades.

In accordance with another aspect, an articulated retractor bladecomprises: a proximal segment; a middle segment coupled at a first jointto a distal end of the proximal segment; a distal segment coupled at asecond joint to a distal end of the middle segment; a first actuatorconfigured to cause rotation of the middle segment about the firstjoint; and a second actuator configured to cause rotation of the distalsegment about the second joint. In some embodiments, the first actuatorcomprises a first internal cable extending through the proximal segment,across the first joint, and into the middle segment. In someembodiments, proximal movement of the first internal cable causes themiddle segment to rotate about the first joint. In some embodiments, thearticulated retractor blade further comprises a retention mechanismconfigured to releasably retain the position of the first internalcable. In some embodiments, the retention mechanism comprises a pinionkey coupled to a ratchet. In some embodiments, the second actuatorcomprises a second internal cable extending through the proximalsegment, across the first joint, and into the middle segment, across thesecond joint, and into the distal segment. In some embodiments, proximalmovement of the first internal cable causes the distal segment to rotateabout the second joint. In some embodiments, the articulated retractorblade comprises a retention mechanism configured to releasably retainthe position of the first internal cable. In some embodiments, theretention mechanism comprises a pinion key coupled to a ratchet. In someembodiments, the articulated retractor blade comprises at least onecamera disposed on a surface of the middle segment.

In accordance with another aspect, a method comprises: positioning aretractor stage over an opening in a body, wherein the retractor stagecomprises a ring defining an aperture, the ring substantially alignedwith a first plane substantially orthogonal to a first axis; arrangingat least one blade coupled to said stage and positioned within saidcentral aperture, the blade away from said plane and into the opening;positioning the at least one blade rotationally and radially withrespect to said ring such that a surface of the blade abuts an edge ofthe opening; and tilting the at least one blade with respect to thefirst axis; and inserting a surgical tool at least partially through theaperture and into the opening in the body. In some embodiments, themethod further comprises: arranging a plurality of blades coupled tosaid stage and positioned within said central aperture, each of theblades extending away from said plane and into the opening; positioningeach of the blades rotationally and radially with respect to said ringsuch that a surface of each blade abuts an edge of the opening; andtilting each of the blades with respect to the first axis. In someembodiments, tilting the at least one blade comprises flexing the blade.In some embodiments, the at least one blade is jointed, and whereintilting the at least one blade comprises bending the blade at a joint.In some embodiments, the at least one blade comprises: a proximalsegment; a middle segment coupled at a first joint to a distal end ofthe proximal segment; a distal segment coupled at a second joint to adistal end of the middle segment; a first actuator configured to causerotation of the middle segment about the first joint; and a secondactuator configured to cause rotation of the distal segment about thesecond joint. In some embodiments, the at least one blade comprises atleast one camera disposed therein. In some embodiments, the methodfurther comprises obtaining an image from the camera and displaying saidimage.

In accordance with another aspect, a retractor comprises: a main body; afirst blades comprising clip-on fastener for removable attaching saidfirst blade to said main body, a second blades comprising clip-onfastener for removable attaching said second blade to said main body, atleast one camera connected to at least said first blade. In someembodiments, at lease one of said first and second blades are flexible.In some embodiments, said first and second blades have differentdimensions. In some embodiments, said first and second blades havedifferent stiffness.

In accordance with another aspect, a retractor comprises: a main body; afirst blades comprising clip-on fastener for removable attaching saidfirst blade to said main body, a second blades comprising clip-onfastener for removable attaching said second blade to said main body,wherein at least one of said first and second blades have differentdimensions, stiffness, or both. In some embodiments, at least one ofsaid first and second blades are flexible.

In accordance with another aspect, a surgical visualization systemcomprises: a retractor; and a plurality of cameras disposed on theretractor, said cameras producing respective images, wherein the camerasare fastened to the retractor using a clip-on fastener. In someembodiments, said clip-on fastener comprises a clip, a snap, a screw, abolt, a nut, or magnet.

In accordance with another aspect, a clip-on camera system for clippingon a retractor comprises: a plurality of camera modules comprising aplurality of support platforms and at least one camera disposed on thesupport platforms; a fastened configured to clip-on the support platformonto the retractor; electrical signal lines from the camera; anelectrical connector electrically connected to the electrical sensor;and a central bus box for receiving the plurality of electrical linesand connectors. In some embodiments, said clip-on fastener comprises aclip, a snap, a screw, a bolt, a nut, or magnet. In some embodiments,said electrical signal lines are between about 1 to 4 inches long.

In accordance with another aspect, a surgical visualization systemcomprises: a retractor configured to provide access to a surgical site;a plurality of cameras configured to acquire video images of thesurgical site, at least one of the plurality of cameras being disposedon the retractor and configured to acquire video images within anopening to which the retractor provides access; and an image processingsystem in communication with the plurality of cameras, the imageprocessing system comprising at least one physical processor, whereinthe image processing system is configured to: receive the video imagesacquired by the plurality of cameras; receive input from a userindicating a selection of at least two of the plurality of cameras, saidselection being less than all of said cameras in said plurality ofcameras; provide output video images based on the video images acquiredby the selected cameras, the output video images being provided forsimultaneous viewing; and resize or arrange the simultaneously viewableoutput video images from the selected cameras to present them on adisplay according to received input. In some embodiments, the imageprocessing system is configured to increase a size of a first one of thesimultaneously viewable output video images in relation to a second oneof the simultaneously viewable output video images based at least partlyon received input. In some embodiments the image processing system isconfigured to arrange a first one of the simultaneously viewable outputvideo images in a more central location in relation to a second one ofthe simultaneously viewable output video based at least partly onreceived input. In some embodiments at least one of the simultaneouslyviewable output video images are represented by a reduced-size real-timevideo stream that is configured to be presented on a graphical userinterface for selection by a user, wherein the graphical user interfaceincludes a representation of a position of the retractor and theplurality of cameras. In some embodiments at least one of thesimultaneously viewable output video images is represented by areduced-size real-time video stream that is configured to be presentedon a display for selection by a user, the reduced-size real-time videostream comprising video from the respective camera. In some embodimentsthe image processing system is (a) configured to display video imagesthat are from a camera that is not selected and that are not displayedon the display after receiving input from the user indicating aselection thereof, (b) is configured to display video images that arefrom the camera that is not selected and that are displayed as areduced-size real-time video stream more prominently after receivinginput from the user indicating a selection thereof, or (c) configured asset forth in both (a) and (b). In some embodiments the image processingsystem is configured to the output video images in a tiled format. Insome embodiments the tiled output images comprise at least three images.In some embodiments the tiled output images comprise at least fourimages. In some embodiments the image processor is configured to rotatethe tiled output images around a single, common axis. In someembodiments the output video images from at least two of the pluralityof cameras are discontinuous. In some embodiments, the system furthercomprises a second display, wherein the cameras that are not selected bythe user can be displayed on the second display. In some embodiments theplurality of selected cameras comprise at least first and second camerasthat are disposed on the retractor in positions opposite one another. Insome embodiments the image processing system is further configuredrotate video images from said first selected camera 180° with respect tovideo images from said second selected camera.

In accordance with another aspect, a surgical visualization systemcomprises: a retractor configured to provide access to a surgical site;a plurality of cameras configured to acquire video images of thesurgical site, at least one of the plurality of cameras being disposedon the retractor and configured to acquire video images of said surgicalsite; and an image processing system in communication with the pluralityof cameras, the image processing system comprising at least one physicalprocessor, wherein the image processing system is configured to: receivethe video images acquired by the plurality of cameras; receive inputindicating a selection of video images from a camera; and providingoutput video images based on video images acquired with the selectedcamera, wherein the image processing system is configured to presentoutput video images from a first camera, and wherein the imageprocessing system is further configured to swap the presentation of theoutput video images of the first camera with video images from a secondcamera in response to a request to resize the output video images fromthe second camera to be larger than a threshold size. In someembodiments, the image processing system is configured to recognizeenlargement by the user of a reduced-size real-time video stream beyondthe threshold value as the request to resize the output video imagesfrom the second camera. In some embodiments, the image processing systemis configured to convert the first image to a reduced-size real-timevideo stream as part of swapping the presentation of the video imagesfrom the second camera with the video images of the first camera. Insome embodiments, in response to selection by a user, the imageprocessing system is configured to present output video images from athird camera positioned over output video images from the second camera,wherein the output images from the third camera are less than thethreshold value. In some embodiments, in response to selection by a userafter the presentation of the output video images of the first camera isswapped with video images from a second camera, the image processingsystem is configured to present output video images from the firstcamera positioned over output video images from the second camera,wherein the output images from the first camera are less than thethreshold value. In some embodiments, the system further comprises adisplay in communication with said image processing system, said displayconfigured to display said video images from said first and secondcameras, wherein said threshold value is at least 70% and less than 85%of the size of the display. In some embodiments, said threshold value isat least 85% of the size of the display. In some embodiments, the systemfurther comprises a second display, wherein the cameras that are notselected by the user can be displayed on the second display. In someembodiments, the second display presents a graphical user interface forconfiguration of the output video images on the display.

In accordance with another aspect, a surgical visualization system cancomprise: a retractor configured to provide an opening in a surgicalsite; a plurality of cameras configured to acquire video images of thesurgical site, at least one of the plurality of cameras being disposedon the retractor and configured to acquire video images within theopening provided by the retractor; and an image processing system incommunication with the plurality of cameras, the image processing systemcomprising at least one physical processor, wherein the image processingsystem is configured to: receive the video images acquired by theplurality of cameras; provide representations of each of the pluralityof cameras by a camera icon on a display, each camera icon presenting areal-time representation of the video images acquired by the cameraassociated with the camera icon; receive input indicating a selection ofa camera based on a selection of the associated camera icon; andproviding output video images based on video images acquired with theselected camera, the output video images initially arranged in a mannerconsistent with a physical arrangement of the selected camera. In someembodiments, the image processing system can be further configured to:receive input indicating a size with which to present output videoimages acquired with a designated camera, a relative position to presentthe output video images acquired with the designated camera, or both;and providing the output images of the designated camera according tothe received input. In some embodiments, in response to selection by auser, the image processing system can be configured to present outputvideo images from a first camera positioned over output video imagesfrom a second, wherein the output images from the second camera islarger than the output imagery from the first camera. In someembodiments, a the image processing system can be further configured toswap the presentation of the output video images of the first and secondcameras in response to a request to resize the first output video imagesto be larger than a first threshold size or a request to resize thesecond output video images to be smaller than a second threshold size.In some embodiments, the image processor can be further configured torotate the output video images of the first and second cameras around asingle, common axis.

In accordance with another aspect, a surgical visualization systemcomprises: a retractor; a plurality of cameras, at least one of saidcameras being disposed on said retractor; and an image processing modulein communication with said cameras, said image processing moduleconfigured to arranged a plurality of video images for simultaneousviewing on a display in a tiled format, said tiled video images beingsuperimposed over a larger video image from one of said cameras. In someembodiments, said larger video image comprises a wider field-of-viewimage and said tiled video images comprise narrower field-of-viewimages. In some embodiments, said wide field-of-view image has a fieldat least 1.3 times larger than one of said tiled video images. In someembodiments, said wide field-of-view image has a field at least 1.5times larger than one of said tiled video images. In some embodiments,said wide field-of-view image has a field at least 1.75 times largerthan one of said tiled video images. In some embodiments, said widefield-of-view image has a field at least 2.0 times larger than one ofsaid tiled video images. In some embodiments, at least two of saidcameras are disposed on said retractor. In some embodiments, whereinsaid plurality of cameras are disposed on said retractor, at least oneof said cameras being disposed on said retractor. In some embodiments,at least one of said cameras is disposed on a surgical tool.

In accordance with another aspect, a surgical visualization systemcomprises: a retractor; at least one cameras disposed on said retractor;and an image processing module in communication with said camera, saidimage processing module configured to provide a main video image on adisplay, said image processing module further configured to receive avideo image from a camera disposed on a surgical tool and superimposesaid video image from said surgical tool camera on said main videoimage, wherein said main video image is larger than said video imagefrom said surgical tool cameras.

In accordance with another aspect, a surgical visualization systemcomprises: a retractor; at least one cameras disposed on said retractor;and an image processing module in communication with said camera, saidimage processing module configured to provide a main video image on adisplay, said image processing module further configured to receive avideo image from a camera disposed on a surgical tool and superimposesaid video image from said surgical tool camera on said main videoimage, wherein said main video image is larger than said video imagefrom said surgical tool cameras. In some embodiments, said main videoimage comprises a relatively wide field-of-view image in comparison tosaid video image from said camera on said surgical tool which comprise arelatively narrow field-of-view image. In some embodiments, said mainfield-of-view image has a field at least 1.5 times larger than saidvideo images from said camera on said surgical tool. In someembodiments, said main field-of-view image has a field at least 1.75times larger than said video images from said camera on said surgicaltool. In some embodiments, said main field-of-view image has a field atleast 2.0 times larger than said video images from said camera on saidsurgical tool. In some embodiments, said main field-of-view image has afield at least 2.3 times larger than said video images from said cameraon said surgical tool. In some embodiments, at least two of said camerasare disposed on said retractor. In some embodiments, said plurality ofcameras are disposed on said retractor, at least one of said camerasbeing disposed on said retractor. In some embodiments, at least one ofsaid cameras is disposed on a surgical tool.

In accordance with another aspect, some embodiments provide for asurgical visualization system that includes a retractor configured toprovide an opening in a surgical site. The surgical visualization systemincludes a plurality of cameras configured to acquire video images ofthe surgical site, at least one of the plurality of cameras beingdisposed on the retractor and configured to acquire video images withinthe opening provided by the retractor. The surgical visualization systemincludes an image processing system in communication with the pluralityof cameras, the image processing system comprising at least one physicalprocessor. The image processing system is configured to receive thevideo images acquired by the plurality of cameras; receive input from auser indicating a selection of at least two of the plurality of cameras;provide output video images based on the video images acquired by theselected cameras, the output video images being provided forsimultaneous viewing; and resize or arrange the simultaneously viewableoutput video images from the selected cameras to present them moreprominently on a display in comparison to output video images based onthe video images acquired by any of the plurality of cameras which isnot selected. In some aspects, the image processing system is configuredto increase a size of the simultaneously viewable output video images inrelation to the output video images from the non-selected cameras. Theimage processing system can also be configured to arrange thesimultaneously viewable output video images in a more central locationin relation to the output video images from the non-selected cameras. Insome implementations, the simultaneously viewable output video imagesinclude icons that are configured to be presented on a display forselection by a user. The icons can include images from the respectivecameras.

In some embodiments, the image processing system is configured toarrange the output video images from the selected cameras in a mannerconsistent with a physical arrangement of the selected cameras. Thephysical arrangement of the selected cameras can include a location ofthe selected cameras, a field-of-view of the selected cameras, or both.In some aspects, the image processing system is configured to the outputvideo images in a tiled format. The tiled output images can be arrangedin a geometrical arrangement consistent with locations of the pluralityof cameras with respect to each other, with orientations of theplurality of cameras with respect to each other, or both. The tiledoutput images can include in some aspects at least four images. In someimplementations, the image processor is configured to rotate the tiledoutput images around a single, common axis. In some aspects, the outputvideo images from at least two of the plurality of cameras arediscontinuous.

In some embodiments, at least two of the plurality of cameras can bedisposed on the retractor positioned opposite one another. The imageprocessing system can be further configured to output video images fromthe at least two cameras disposed on the retractor that are rotated 180degrees with respect to one another. The image processing system can befurther configured to arrange the output video images from the selectedcameras in a manner consistent with a location of the at least twocameras disposed on the retractor, a field-of-view of the at least twocameras disposed on the retractor, or both.

In accordance with another aspect, some embodiments provide for asurgical visualization system that includes a retractor configured toprovide an opening in a surgical site. The surgical visualization systemincludes a plurality of cameras configured to acquire video images ofthe surgical site, at least one of the plurality of cameras beingdisposed on the retractor and configured to acquire video images withinthe opening provided by the retractor. The surgical visualization systemincludes an image processing system in communication with the pluralityof cameras, the image processing system comprising at least one physicalprocessor. The image processing system is configured to receive thevideo images acquired by the plurality of cameras; to providerepresentations of each of the plurality of cameras by a camera icon ona display, each camera icon presenting a real-time representation of thevideo images acquired by the camera associated with the camera icon; toreceive input indicating a selection of a camera based on a selection ofthe associated camera icon; and to provide output video images based onvideo images acquired with the selected camera, the output video imagesinitially arranged in a manner consistent with a physical arrangement ofthe selected camera. In a further aspect, the image processing system isconfigured to receive input indicating a size with which to presentoutput video images acquired with a designated camera, a relativeposition to present the output video images acquired with the designatedcamera, or both; and to providing the output images of the designatedcamera according to the received input. In some implementations, inresponse to selection by a user, the image processing system isconfigured to present output video images from a first camera positionedover output video images from a second, wherein the output images fromthe second camera is larger than the output imagery from the firstcamera. The image processing system can be further configured to swapthe presentation of the output video images of the first and secondcameras in response to a request to resize the first output video imagesto be larger than a first threshold size or a request to resize thesecond output video images to be smaller than a second threshold size.In some aspects, the image processor is configured to rotate the outputvideo images of the first and second cameras around a single, commonaxis.

In accordance with another aspect, a surgical visualization system cancomprise: a retractor configured to provide an opening in a surgicalsite; a plurality of cameras configured to acquire video images of thesurgical site, at least one of the plurality of cameras being disposedon the retractor and configured to acquire video images within theopening provided by the retractor; and an image processing system incommunication with the plurality of cameras, the image processing systemcomprising at least one physical processor, wherein the image processingsystem is configured to: receive the video images acquired by theplurality of cameras; receive input from a user indicating a selectionof at least two of the plurality of cameras; provide output video imagesbased on the video images acquired by the selected cameras, the outputvideo images being provided for simultaneous viewing; and resize orarrange the simultaneously viewable output video images from theselected cameras to present them more prominently on a display incomparison to output video images based on the video images acquired byany of the plurality of cameras which is not selected. In someembodiments, the image processing system is configured to arrange theoutput video images from the selected cameras in a manner consistentwith a physical arrangement of the selected cameras. According to somevariants, the physical arrangement of the selected cameras comprises alocation of the selected cameras, a field-of-view of the selectedcameras, or both. The image processing system can configured to increasea size of the simultaneously viewable output video images in relation tothe output video images from the non-selected cameras. In someembodiments, the image processing system is configured to arrange thesimultaneously viewable output video images in a more central locationin relation to the output video images from the non-selected cameras.The simultaneously viewable output video images can comprise icons thatare configured to be presented on a display for selection by a user. Insome embodiments, the icons comprise images from the respective cameras.

In accordance with another aspect, a surgical visualization system cancomprise: a retractor configured to provide an opening in a surgicalsite; a plurality of cameras, each camera having a field of view andconfigured to acquire video images of a portion of the surgical sitecorresponding to the field of view, at least one of the plurality ofcameras being disposed on the retractor and configured to acquire videoimages within the opening provided by the retractor; and an imageprocessing system in communication with the plurality of cameras, theimage processing system comprising at least one physical processor,wherein the image processing system is configured to: receive the videoimages acquired by the plurality of cameras; for each of the pluralityof cameras, provide output video images based on the acquired videoimages, the output video images being provided for simultaneous viewing;and arrange each of the output video images from the plurality ofcameras in a manner consistent with their respective fields of view. Theimage processing system can be configured to the output video images ina tiled format. In some embodiments, the tiled output images arearranged in a geometrical arrangement consistent with locations of theplurality of cameras with respect to each other, with orientations ofthe plurality of cameras with respect to each other, or both. The tiledoutput images can comprise at least four images. In some embodiments,the image processor is configured to rotate the tiled output imagesaround a single, common axis. The output video images from at least twoof the plurality of cameras can be discontinuous.

According to some variants, the plurality of cameras are eachrepresented by an icon that is configured to be presented on a displayfor selection by a user. The icons can be configured to present areal-time representation of the video images acquired by the respectivecamera. In some embodiments, the image processing system is configuredto receive input from a user indicating a size with which to presentoutput video images acquired with the selected camera, a relativeposition to present the output video images acquired with the selectedcamera, or both. Some embodiments can be configured such that, inresponse to selection by a user, the image processing system isconfigured to present output video imagery from a first camerapositioned over output video imagery from a second, wherein the outputimagery from the second camera is larger than the output imagery fromthe first camera. The image processing system can be configured to swapthe presentation of the output video imagery of the first and secondcameras in response to a request to resize the first video output to belarger than a first threshold size or a request to resize the secondvideo output to be smaller than a second threshold size. In someembodiments, at least two of the plurality of cameras are disposed onthe retractor positioned opposite one another. In some embodiments, theimage processing system is configured to output video images from the atleast two cameras disposed on the retractor that are rotated 180 degreeswith respect to one another.

In accordance with another aspect, a medical apparatus comprises: aretractor configured to provide access to a surgical site; a pluralityof cameras configured to acquire video images of the surgical site, atleast one of the plurality of cameras being disposed on the retractorand configured to acquire video images within the opening provided bythe retractor; a binocular viewing assembly comprising a housing and aplurality of oculars, the plurality of oculars configured to provideviews of at least one display disposed in the housing; a viewingarticulating arm, the binocular viewing assembly disposed on the viewingarticulating arm, the viewing articulating arm configured to adjust aposition of the binocular viewing assembly; a support, the viewingarticulating arm attached to the support such that the viewingarticulating arm can move relative to the support; and an imageprocessing system in communication with the plurality of cameras and theat least one display, the image processing system comprising at leastone physical processor, wherein the image processing system isconfigured to: receive video images acquired by the plurality ofcameras, provide output video images based on the received video images,and present the output video images on the at least one display so thatthe output video images are viewable through the plurality of oculars,wherein the binocular viewing assembly does not provide a view of thesurgical site through the oculars via an optical pathway that passesthrough the housing. In some embodiments, the image processing system isconfigured to provide 3-D viewing of camera images through thebinoculars. In some embodiments, the apparatus further comprises anauxiliary camera disposed on the binocular viewing assembly, the camerahaving a field of view that can be configured to include the surgicalsite, wherein the camera is configured to provide a surgical microscopeview of the surgical site. In some embodiments, the auxiliary cameradisposed on the binocular viewing assembly comprises an optical assemblyproviding an adjustable working distance of between about 15 cm andabout 45 cm. In some embodiments, the optical assembly has a variablemagnification of between about −0.5× and about 10×. In some embodiments,the medical apparatus further comprises a second articulating arm; andan auxiliary camera disposed on the second articulated arm, theauxiliary camera having a field of view that can be configured toinclude the surgical site wherein the camera is configured to provide asurgical microscope view of the surgical site, wherein the imageprocessing system is configured to display the surgical microscope viewon the at least one display. In some embodiments, the auxiliary cameradisposed on the camera platform comprises an optical assembly providingan adjustable working distance of between about 15 cm and about 45 cm.In some embodiments, the optical assembly has a variable magnificationof between about −0.5× and about 10×.

In accordance with another aspect, a medical apparatus comprises: aretractor configured to provide access to a surgical site; a pluralityof cameras configured to acquire video images of the surgical site, atleast one of the plurality of cameras being disposed on the retractorand configured to acquire video images within the opening provided bythe retractor; an auxiliary camera having a field of view that can beconfigured to include the surgical site, wherein the auxiliary camera isconfigured to provide a surgical microscope view of the surgical site; abinocular viewing assembly comprising a housing and a plurality ofoculars, the plurality of oculars configured to provide views of atleast one display disposed in the housing; a viewing articulating arm,the binocular viewing assembly disposed on the viewing articulating arm,the viewing articulating arm configured to adjust a position of thebinocular viewing assembly; a support, the viewing articulating armattached to the support such that the viewing articulating arm can moverelative to the support; and an image processing system in communicationwith the plurality of cameras, the auxiliary camera, and the at leastone display, the image processing system comprising at least onephysical processor, wherein the image processing system is configuredto: receive video images acquired by the plurality of cameras, receivevideo images acquired by the auxiliary camera, provide output videoimages based on the received video images, present the output videoimages on the at least one display so that the output video images areviewable through the plurality of oculars, and switch between displayingthe output video images comprising at least one output video image fromthe auxiliary camera and at least one output video image from the atleast one camera disposed on the retractor, wherein the binocularviewing assembly does not provide a view of the surgical site throughthe oculars via an optical pathway that passes through the housing. Insome embodiments, the auxiliary camera comprises a 3-D camera and thedisplays are configured to provide 3-D viewing of images from the 3-Dcameras. In some embodiments, the auxiliary camera comprises an opticalassembly having an adjustable working distance of between about 15 cmand about 45 cm. In some embodiments, the optical assembly has avariable magnification of between about −0.5× and about 10×. In someembodiments, the auxiliary camera is configured to provide views of thesurgical site from a distance further than the cameras disposed on theretractor.

In accordance with another aspect, a medical apparatus comprises aretractor configured to provide access to a surgical site; a pluralityof cameras configured to acquire video images of the surgical site, atleast one of the plurality of cameras being disposed on the retractorand configured to acquire video images within the opening provided bythe retractor; a viewing assembly comprising a housing and at least onedisplay within the housing, the at least one display being configured toprovide images from the plurality of cameras; at least one virtualdisplay input device configured to acquire input from a user of themedical apparatus; and an image processing system in communication withthe plurality of cameras, the at least one virtual display camera, andthe at least one display, the image processing system comprising atleast one physical processor, wherein the image processing system isconfigured to: receive video images acquired by the plurality ofcameras, provide output video images based on the received video images,detect the input from the user, interpret the input from the user, andprovide a virtual touchscreen wherein interaction with the virtualtouchscreen is provided through the detected and interpreted user input.In some embodiments, the user input is associated with features of agraphical user interface displayed on the at least one display. In someembodiments, the graphical user interface includes reduced-size videoimage feeds provided by the image processing module. In someembodiments, the viewing assembly comprises a pair of oculars, andoptical paths from the oculars to the at least one display. In someembodiments, the at least one virtual display input device is disposedon the viewing assembly. In some embodiments, the apparatus furthercomprises virtual display sensors configured to provide informationrelated to the user input to the image processing system, theinformation being provided in addition to the data acquired with thevirtual display input device. In some embodiments, the apparatus furthercomprises at least one auxiliary camera configured such that images fromthe at least one auxiliary camera can be viewed on at least one of thedisplays, said auxiliary camera configured to provide a surgicalmicroscope view. In some embodiments, the at least one virtual displayinput device comprises a camera. In some embodiments, the at least onevirtual display input device comprises at least 3 cameras.

In accordance with another aspect, a medical apparatus comprises: aretractor configured to provide an opening in a surgical site; aplurality of cameras configured to acquire video images of the surgicalsite, at least one of the plurality of cameras being disposed on theretractor and configured to acquire video images within the openingprovided by the retractor; a binocular viewing assembly comprising ahousing and a plurality of oculars, the plurality of oculars configuredto provide views of at least one display disposed in the housing; aviewing articulating arm, the binocular viewing assembly disposed on theviewing articulating arm, the viewing articulating arm configured toadjust a position of the binocular viewing assembly; a support stand,the viewing articulating arm attached to the support stand such that theviewing articulating arm can move relative to the support stand; and animage processing system in communication with the plurality of camerasand the at least one display, the image processing system comprising atleast one physical processor, wherein the image processing system isconfigured to: receive video images acquired by the plurality ofcameras, provide output video images based on the received video images,and present the output video images on the at least one display so thatthe output video images are viewable through the plurality of oculars,wherein the binocular viewing assembly does not provide a view of thesurgical site through the oculars via an optical pathway that passesthrough the housing. In some embodiments, the image processing system isconfigured to provide 3-D viewing of camera images through thebinoculars using at least two displays. In some embodiments, theapparatus further comprises a camera disposed on the binocular viewingassembly, the camera having a field of view that can be configured toinclude the surgical site, wherein the camera is configured to provide asurgical microscope view of the surgical site. In some embodiments, thecamera disposed on the binocular viewing assembly comprises an opticalassembly providing an adjustable working distance of between about 15 cmand about 45 cm. In some embodiments, the microscope objectivemicroscope objective has a variable magnification of between about 1×and 6×. In some embodiments, the apparatus further comprises: a secondarticulating arm attached to the support stand; a camera platformattached to the second articulating arm such; and a camera disposed onthe camera platform, the camera having a field of view that can beconfigured to include the surgical site wherein the camera is configuredto provide a surgical microscope view of the surgical site, wherein thesecond articulating arm is adjustable independent of the viewingarticulating arm, and wherein the image processing system is configuredto display the surgical microscope view on the at least one display. Insome embodiments, the camera disposed on the camera platform comprisesan optical assembly providing an adjustable working distance of betweenabout 15 cm and about 45 cm. In some embodiments, the microscopeobjective microscope objective has a variable magnification of betweenabout 1× and 6×.

In accordance with another aspect, a medical apparatus comprises: aretractor configured to provide an opening in a surgical site; aplurality of cameras configured to acquire video images of the surgicalsite, at least one of the plurality of cameras being disposed on theretractor and configured to acquire video images within the openingprovided by the retractor; a surgical microscope camera having a fieldof view that can be configured to include the surgical site, wherein thesurgical microscope camera is configured to provide a surgicalmicroscope view of the surgical site; a binocular viewing assemblycomprising a housing and a plurality of oculars, the plurality ofoculars configured to provide views of at least one display disposed inthe housing; a viewing articulating arm, the binocular viewing assemblydisposed on the viewing articulating arm, the viewing articulating armconfigured to adjust a position of the binocular viewing assembly; asupport stand, the viewing articulating arm attached to the supportstand such that the viewing articulating arm can move relative to thesupport stand; and an image processing system in communication with theplurality of cameras, the surgical microscope camera, and the at leastone display, the image processing system comprising at least onephysical processor, wherein the image processing system is configuredto: receive video images acquired by the plurality of cameras, receivevideo images acquired by the surgical microscope camera, provide outputvideo images based on the received video images, and present the outputvideo images on the at least one display so that the output video imagesare viewable through the plurality of oculars, the output video imagescomprising at least one output video image from the surgical microscopecamera being presented with at least one output video image from the atleast one camera disposed on the retractor, wherein the binocularviewing assembly does not provide a view of the surgical site throughthe oculars via an optical pathway that passes through the housing. Insome embodiments, the surgical microscope camera comprises a pair of 3-Dcameras and the displays are configured to provide 3-D viewing of imagesfrom the pair of 3-D cameras. In some embodiments, the surgicalmicroscope camera comprises a microscope objective having an adjustableworking distance of between about 15 cm and about 45 cm. In someembodiments, the microscope objective microscope objective has avariable magnification of between about 1× and 6×. In some embodiments,the surgical microscope camera is configured to provide views of thesurgical site from a distance further than the at least one of theplurality of cameras disposed on the retractor.

In accordance with another aspect, a medical apparatus comprises: aretractor configured to provide an opening in a surgical site; aplurality of cameras configured to acquire video images of the surgicalsite, at least one of the plurality of cameras being disposed on theretractor and configured to acquire video images within the openingprovided by the retractor; a viewing assembly comprising a housing andat least one display attached to the housing, the at least one displaybeing configured to provide images from the plurality of cameras; atleast one virtual display camera configured such that images from the atleast one virtual display camera can be viewed on at least one of thedisplays, the at least one virtual display camera providing views ofgestures by a viewer of the display; and an image processing system incommunication with the plurality of cameras, the at least one virtualdisplay camera, and the at least one display, the image processingsystem comprising at least one physical processor, wherein the imageprocessing system is configured to: receive video images acquired by theplurality of cameras, provide output video images based on the receivedvideo images, and provide a virtual touchscreen by detecting thegestures made by the viewer imaged by the virtual image camera. In someembodiments, the gestures are associated with imaged features displayedon the at least one display. In some embodiments, the image featurescomprise icons provided by the image processing module. In someembodiments, the viewing assembly comprises a housing, a pair ofoculars, and optical paths from the oculars to the at least one display,the at least one display disposed in the housing. In some embodiments,the at least one virtual display camera is disposed on the viewingassembly. In some embodiments, the at least one viewing assembly isdisposed on an articulating arm. In some embodiments, the apparatusfurther comprises at least one surgical microscope camera configuredsuch that images from the at least one surgical microscope camera can beviewed on at least one of the displays.

In accordance with another aspect, a surgical system can comprise: oneor more surgical tools that can be configured to be powered by hydraulicfluid; a hydraulic pressure system fluidly connected to the one or moresurgical tools, the hydraulic pressure system can be configured topressurize the hydraulic fluid, the hydraulic pressure system cancomprise a first hydraulic pressure source having a first hydraulicfluid chamber in selective fluid communication with the one or moresurgical tools, the first hydraulic pressure source having a compressionstroke in which the first hydraulic pressure source increases thepressure of the hydraulic fluid within the first hydraulic fluid chamberand an expansion stroke in which the first hydraulic pressure sourcedecreases the pressure of the hydraulic fluid within the first hydraulicfluid chamber; a second hydraulic pressure source having a secondhydraulic fluid chamber in selective fluid communication with the one ormore surgical tools, the second hydraulic pressure source having acompression stroke in which the second hydraulic pressure sourceincreases the pressure of the hydraulic fluid within the secondhydraulic fluid chamber and an expansion stroke in which the secondhydraulic pressure source decreases the pressure of the hydraulic fluidwithin the second hydraulic fluid chamber, wherein the second hydraulicpressure source can be configured to operate its compression stroke whenthe first hydraulic pressure source operates its expansion stroke andwherein the second hydraulic pressure source can be configured tooperates its expansion stroke when the first hydraulic pressure sourceoperates its compression stroke; and one or more valves positioned in afluid path between the one or more surgical tools and one or more of thefirst hydraulic pressure source and the second hydraulic pressuresource, the one or more valves can be configured to selectively closeand open fluid communication between the one or more surgical tools andone or more of the first hydraulic pressure source and the secondhydraulic pressure source; and one or more hydraulic fluid sources inselective fluid communication with one or more of the first hydraulicpressure source and the second hydraulic pressure source. In someembodiments, a surgical system can further comprise a first pneumaticpressure source, the first pneumatic pressure source can comprise: apneumatic fluid chamber; a piston positioned within the fluid chamber;and an actuator operably connected to the piston, the actuator can beconfigured to move the piston in a compression stroke and an expansionstroke; wherein the pneumatic fluid chamber can be in selective fluidcommunication with one or more of the first hydraulic fluid chamber andthe second hydraulic fluid chamber. In some embodiments, the surgicalsystem can include a pneumatic pump in selective fluid communicationwith the pneumatic fluid chamber. In some embodiments, the surgicalsystem can further comprise a hydraulic turbine that can be configuredto actuate one or more of the one or more surgical tools, the hydraulicturbine in selective fluid communication with the hydraulic pressuresystem. In some embodiments, the hydraulic turbine can be powered bypressurized hydraulic fluid from the hydraulic pressure system. In someembodiments, at least a portion of the pressurized hydraulic fluid usedto power the hydraulic turbine can be returned to the hydraulic pressuresystem after powering the hydraulic turbine. In some embodiments, thesurgical system can further comprise one or more surgical toolsconfigured to be powered by pneumatic fluid. In some embodiments, thesurgical system can further comprise a pneumatic assembly that can beconfigured to selectively power one or more of the one or more surgicaltools that can be configured to be powered by pneumatic fluid, thepneumatic assembly in selective fluid communication with a pneumaticpump. In some embodiments, the surgical device can be a drill.

In accordance with another aspect, a surgical system can comprise: oneor more surgical tools that can be configured to be powered by hydraulicfluid; a hydraulic pressure system fluidly connected to the one or moresurgical tools, the hydraulic pressure system can be configured topressurize the hydraulic fluid, the hydraulic pressure systemcomprising: a cassette assembly that can have: a cassette housing; afirst hydraulic fluid chamber positioned at least partially within thecassette housing and in selective fluid communication with one or moreof the one or more surgical tools; a plurality of fluid ports positionedon the cassette housing; and one or more valves located in or on thecassette housing; a first hydraulic pressure source fluidly coupled withthe first hydraulic fluid chamber, the first hydraulic pressure sourcehaving a compression stroke in which the first hydraulic pressure sourceincreases the pressure of the hydraulic fluid within the first hydraulicfluid chamber and an expansion stroke in which the first hydraulicpressure source decreases the pressure of the hydraulic fluid within thefirst hydraulic fluid chamber; one or more valves positioned in a fluidpath between the one or more surgical tools and the first hydraulicfluid chamber, the one or more valves configured to selectively closeand open fluid communication between the one or more surgical tools andthe first hydraulic fluid chamber; and one or more hydraulic fluidsources in selective fluid communication with one or more of the firsthydraulic pressure source and the second hydraulic pressure source. Insome embodiments, the cassette assembly can further comprise: a secondhydraulic fluid chamber positioned at least partially within thecassette housing and in selective fluid communication with one or moreof the one or more surgical tools; and a second hydraulic pressuresource fluidly coupled to the second hydraulic fluid chamber, the secondhydraulic pressure source can have a compression stroke in which thesecond hydraulic pressure source increases the pressure of the hydraulicfluid within the second hydraulic fluid chamber and an expansion strokein which the second hydraulic pressure source decreases the pressure ofthe hydraulic fluid within the second hydraulic fluid chamber, whereinthe second hydraulic pressure source can be configured to operate itscompression stroke when the first hydraulic pressure source operates itsexpansion stroke and wherein the second hydraulic pressure source can beconfigured to operates its expansion stroke when the first hydraulicpressure source operates its compression stroke. In some embodiments,the surgical system can further comprise one or more valves positionedin a fluid path between the one or more surgical tools and the secondhydraulic fluid chamber, the one or more valves can be configured toselectively close and open fluid communication between the one or moresurgical tools and the second hydraulic fluid chamber. In someembodiments, the cassette assembly can be disposable. In someembodiments, the surgical tool can be disposable. In some embodiments,the surgical system can further comprise one or more washing nozzles influid communication with one or more of the first hydraulic fluidchambers or the second hydraulic fluid chamber or the hydraulic fluidsource, the one or more washing nozzles can be configured to directhydraulic fluid toward one or more light sources.

In accordance with another aspect, a medical apparatus can comprise: asurgical drill; and a hydraulic motor providing hydraulic power to saiddrill. In some embodiments, the surgical drill can be configured mill.In some embodiments, said hydraulic motor can comprise a turbine connectto a saline supply.

In accordance with another aspect, a hydraulic actuation system cancomprise: a user interface; a drive system in communication with theuser interface; a chamber; an inflatable element at least partially inthe chamber, the inflatable element inflatable upon the user interfacesending a signal to the drive system causing fluid to flow into theinflatable element; and a piston configured to be linearly displaced bythe inflatable element upon inflation of the inflatable element. In someembodiments, the inflatable element can comprise a balloon.

In accordance with another aspect, a medical apparatus can comprise: asurgical tool; and a hydraulic system providing hydraulic power to saidsurgical tool, wherein said hydraulic system can comprise (a) a pumpwith disposable components and (b) disposable lines. In someembodiments, the hydraulic system can comprise a disposable spindlevalve body. In some embodiments, the hydraulic system can comprisedisposable Kerrison balloons. In some embodiments, the surgical tool cancomprise a disposable drill. In some embodiments, the surgical tool canbe disposable and non-autoclavable. In some embodiments, the hydraulicsystem can comprise a disposable and non-autoclavable slave pumpactuator. In some embodiments, the hydraulic system can comprise adisposable and non-autoclavable slave turbine. In some embodiments, thehydraulic system can comprise a disposable and non-autoclavable valve.

In accordance with another aspect, a method of cleaning a hydraulicsystem for coupling to a surgical tool, the method can comprise:flushing fluid through at least part of the hydraulic system; flushingair through said part of the hydraulic system; and sterilization of saidpart of said hydraulic system. In some embodiments, said flushing fluidcan be followed by flushing air. In some embodiments, said flushing aircan be followed by sterilization. In some embodiments, said flushingfluid can be followed by sterilization. In some embodiments, saidflushing fluid can be followed by flushing air and said flushing air canbe followed by sterilization.

In accordance with another aspect, a medical apparatus can comprise: asurgical device; a hydraulic system providing hydraulic power to saidsurgical device; and manifold configured to selectively direct hydraulicfluid to different applications. In some embodiments, said manifold canbe disposable. In some embodiments, said manifold can comprise valves.In some embodiments, said manifold can comprise pumps.

In accordance with another aspect, a surgical tool can comprise: adrill; a hydraulic impeller assembly can comprise: a turbine housingdefining a blade cavity, a flow director positioned at least partiallywithin the turbine housing, an impeller having a plurality of impellerblades, the impeller positioned at least partially within the bladecavity, an output shaft rotatably connected to the impeller, the outputshaft configured to transfer a torque from the impeller to the drill,and one or more ports in a wall of the blade cavity providing fluidcommunication between an interior of the blade cavity and an exterior ofthe blade cavity; a hydraulic pressure source; a return fluid lineconfigured to connect to facilitate fluid communication between the oneor more ports and the hydraulic pressure source; and a vacuum sourceconfigured to extract fluid through the one or more ports from the bladecavity. In some embodiments, the vacuum source can be a pump. In someembodiments, the vacuum source a bypass channel can be configured todirect high velocity fluid past the blade cavity, wherein a pressuredifferential between the high velocity fluid in the bypass channel and alow velocity fluid in the blade cavity draws low velocity fluid throughthe one or more ports from the blade cavity to the bypass channel.

In accordance with another aspect, a medical device system can comprise:a surgical tool; a hydraulic fluid source; a disposable cassettecomprising: a housing; one or more hydraulic fluid chambers housed atleast partially housed within an interior of the housing; a plurality offluid ports positioned on the housing and configured to facilitate fluidcommunication between an exterior of the fluid housing and the interiorof the housing; a plurality of proportional valves positioned on thehousing; and a plurality of fluid channels configured to facilitatefluid communication between the plurality of ports, the one or morehydraulic fluid chambers, and the plurality of proportional valves; atool fluid line fluidly connecting one of the plurality of fluid portsto the surgical tool; and a fluid source line fluidly connecting thehydraulic fluid source to one of the plurality of fluid ports. In someembodiments, one or more of the plurality of proportional valves cancomprise a valve cavity in the housing and a flexible pad sealinglydisposed over the valve cavity. In some embodiments, one or more of theplurality of proportional valves can be actuated by a linearelectromagnetic actuators. In some embodiments, the surgical tool can becontrolled by one or more of the plurality of proportional valves. Insome embodiments, the hydraulic fluid source can be an IV bag. In someembodiments, the hydraulic fluid can be saline. In some embodiments, thehydraulic fluid can be a physiologically compatible fluid. In someembodiments, the hydraulic fluid can be a physiological saline. In someembodiments, the medical device system can further comprise one or moreoptical components. In some embodiments, the medical device system canfurther comprise one or more nozzles in fluid communication with one ormore of the plurality of ports, the one or more nozzles configured todirect a high velocity, low volume flow of hydraulic fluid to the one oroptical components.

In accordance with another aspect, a surgical visualization systemcomprises a binocular viewing assembly comprising a housing and a pairof eyepieces, said eyepieces configured to provide a view of at leastone display disposed in the housing; an optical assembly disposed on thebinocular viewing assembly, the optical assembly configured to provide asurgical microscope view of a surgical site, the optical assemblycomprising at least one auxiliary camera; an articulating arm, thebinocular viewing assembly disposed on the articulating arm, thearticulating arm configured to adjust a position of the binocularviewing assembly and the optical assembly; and an image processingsystem in communication with the optical assembly and the display, theimage processing system comprising at least one physical processor,wherein the image processing system is configured to: receive videoimages acquired by the auxiliary camera, provide output video imagesbased on the received video images, and present the output video imageson the display so that the output video images are viewable through theeyepiece, wherein the optical assembly is configured to provide aworking distance that is adjustable between about 15 cm and about 45 cm.In some embodiments, the optical assembly is mounted to an isocenterpositioning system. In some embodiments, the isocenter positioningsystem comprises elements configured to allow the optical assembly to beadjusted in three-dimensions such that a field of view of the auxiliarycamera always includes a common point. In some embodiments, theisocenter positioning system comprises an isocenter track attached tothe binocular viewing assembly, the isocenter track configured to allowthe auxiliary camera to be moved to a plurality of locations along theisocenter track and to position the auxiliary camera such that at saidplurality of locations said auxiliary camera said remains a fixeddistance away from a common point. In some embodiments, said auxiliarycamera comprises a Greenough configuration. In some embodiments, saidauxiliary camera comprises a pair of optical paths oriented at an anglewith respect to each other that converge at said common pointestablished by said isocenter positioning system. In some embodiments,the system further comprises a virtual touch camera configured to imagea hand of a user, wherein the image processing system is configured toidentify hand gestures based at least partly on the acquired images ofthe hand of the user to allow the user to interact with a graphical userinterface provided on the display. In some embodiments, said handgestures include gesturing with an optical tool held in said hand. Insome embodiments, the system further comprises a virtual touch sensorattached to the binocular viewing assembly, said image processing systembeing configured to use information from the virtual touch sensor inconjunction with image data from the virtual touch camera to identifygestures to allow the user to interact with a graphical user interfaceprovided on the display. In some embodiments, said binocular viewingassembly is configured not to provide a surgical microscope view via anoptical path from said eyepiece through an aperture in said housing. Insome embodiments, said auxiliary includes a turning mirror or turningprism configured to reduce the thickness profile of said opticalassembly. In some embodiments, said auxiliary camera comprises a pair ofoptical paths that do not share a common objective lens. In someembodiments, the system further comprises a virtual touch input deviceconfigured to receive user input, wherein the image processing system isconfigured to identify commands based at least partly on the acquiredinput from the user to allow the user to interact with a graphical userinterface provided on the display through a representation of the user'shand.

In accordance with another aspect, a surgical visualization systemcomprises: A surgical visualization system comprising: a binocularviewing assembly comprising a housing and a plurality of oculars, theplurality of oculars configured to provide display views of at least onedisplay disposed in the housing, the two display views correspondingrespectively to a left-eye view and a right-eye view; an opticalassembly disposed on the binocular viewing assembly, the opticalassembly comprising a left-eye camera and a right-eye camera configuredto provide a stereoscopic surgical microscope view of a surgical site;an articulating arm, the binocular viewing assembly disposed on thearticulating arm, the articulating arm configured to adjust a positionof the binocular viewing assembly and the optical assembly; and an imageprocessing system in communication with the optical assembly and the atleast one display, the image processing system comprising at least onephysical processor, wherein the image processing system is configuredto: receive video images acquired by the left-eye camera and theright-eye camera, provide output video images based on the receivedvideo images, and present the left-eye output video images via theleft-eye display view and the right-eye output video images via theright-eye display view so that the output video images are viewablethrough the plurality of oculars, wherein the optical assembly providesa convergence angle, the convergence angle being an angle between aleft-eye optical path and a right-eye optical path at the surgical site.In some embodiments, the optical assembly is configured to provide asubstantially constant convergence angle with changing working distance.In some embodiments, the left-eye camera comprises: a left-eye turningprism configured to direct light from the surgical site along a left-eyelens path; a left-eye lens assembly configured to receive the directedlight from the left-eye turning prism and to create a left-eye image; aleft-eye image sensor configured to capture the left-eye image andgenerate a left-eye video image. In some embodiments, the right-eyecamera comprises: a right-eye turning prism configured to direct lightfrom the surgical site along a right-eye lens path; a right-eye lensassembly configured to receive the directed light from the right-eyeturning prism and to create a right-eye image; a right-eye image sensorconfigured to capture the right-eye image and generate a right-eye videoimage. In some embodiments, the left-eye camera and the right-eye cameraare configured to acquire video images of the surgical site at aconvergence point and wherein a distance from the binocular viewingassembly to the convergence point comprises a working distance. In someembodiments, the optical assembly is configured to provide an adjustableworking distance between about 15 cm and about 45 cm. In someembodiments, the left-eye camera comprises a left-eye turning prism andthe right-eye camera comprises a right-eye turning prism, the left-eyeturning prism and the right-eye turning prism being configured to changetheir relative orientations thereby changing the convergence angle toprovide the adjustable working distance. In some embodiments, theoptical assembly is configured to provide a substantially constantconvergence angle with changing working distance. In some embodiments,the left-eye camera and the right-eye camera are configured to adjusttheir relative orientation and position to provide the substantiallyconstant convergence angle. In some embodiments, the left-eye cameracomprises a left-eye prism assembly and the right-eye camera comprises aright-eye prism assembly, the left-eye prism assembly and the right-eyeprism assembly being configured to adjust their relative orientation andposition to provide the substantially constant convergence angle,wherein other elements of the left-eye camera and the other elements ofthe right-eye camera remain substantially stationary. In someembodiments, the optical assembly is configured to provide asufficiently narrow convergence angle to provide stereoscopic imagerythrough an insertion tube. In some embodiments, the insertion tube has awidth between about 25 mm and about 50 mm. In some embodiments, thesufficiently narrow convergence angle is also substantially constantwith changes in working distance.

In accordance with another aspect, a surgical visualization system cancomprise: a binocular viewing assembly comprising a housing and aneyepiece, the eyepiece configured to provide a view of a displaydisposed in the housing; an optical assembly disposed on the binocularviewing assembly, the optical assembly configured to provide a surgicalmicroscope view of a surgical site, the optical assembly comprising asurgical microscope camera; an articulating arm, the binocular viewingassembly disposed on the articulating arm, the articulating armconfigured to adjust a position of the binocular viewing assembly andthe optical assembly; and an image processing system in communicationwith the optical assembly and the display, the image processing systemcomprising at least one physical processor, wherein the image processingsystem is configured to: receive video images acquired by the surgicalmicroscope camera, provide output video images based on the receivedvideo images, and present the output video images on the display so thatthe output video images are viewable through the eyepiece, wherein theoptical assembly is configured to provide a working distance that isadjustable between about 15 cm and about 45 cm. In some embodiments, theoptical assembly can be mounted to an isocenter positioning system. Insome embodiments, the isocenter positioning system can comprise elementsconfigured to allow the optical assembly to be adjusted inthree-dimensions such that a field of view of the surgical microscopecamera always includes a common point. In some embodiments, theisocenter positioning system can comprise an isocenter track attached tothe binocular viewing assembly, the isocenter track configured to allowthe surgical microscope camera to be moved to any location along theisocenter track, wherein the isocenter track orients the surgicalmicroscope camera such that a field of view of the surgical microscopecamera includes a common point at any location along the track. In someembodiments, the surgical visualization can further comprise a virtualtouch camera attached to the binocular viewing assembly, the virtualtouch camera configured to image a hand of a user to allow the user tointeract with a graphical user interface provided on the display. Insome embodiments, the surgical microscope camera can be also the virtualtouch camera. In some embodiments, the surgical visualization canfurther comprise a: a retractor configured to provide an opening in asurgical site; and a plurality of cameras configured to acquire videoimages of the surgical site, at least one of the plurality of camerasbeing disposed on the retractor and configured to acquire video imageswithin the opening provided by the retractor, wherein the imageprocessing system is further configured to: receive video imagesacquired by the plurality of cameras disposed on the retractor; provideoutput images based on the received video images from the plurality ofcameras; and present the output images from the surgical microscopecamera, the output images from the plurality of cameras disposed on theretractor, or a combination of output images from the surgicalmicroscope camera and at least one of the plurality of cameras disposedon the retractor. In some embodiments, the image processor system can befurther configured to receive input indicating which output video imagesto display.

In accordance with another aspect, a surgical visualization system cancomprise: a binocular viewing assembly comprising a housing and aplurality of oculars, the plurality of oculars configured to provideviews of two displays disposed in the housing, the two displayscorresponding respectively to a left-eye display and a right-eyedisplay; an optical assembly disposed on the binocular viewing assembly,the optical assembly comprising a left-eye camera and a right-eye cameraconfigured to provide a stereoscopic surgical microscope view of asurgical site; an articulating arm, the binocular viewing assemblydisposed on the articulating arm, the articulating arm configured toadjust a position of the binocular viewing assembly and the opticalassembly; and an image processing system in communication with theoptical assembly and the two displays, the image processing systemcomprising at least one physical processor, wherein the image processingsystem is configured to: receive video images acquired by the left-eyecamera and the right-eye camera, provide output video images based onthe received video images, and present the left-eye output video imageson the left-eye display and the right-eye output video images on theright-eye display so that the output video images are viewable throughthe plurality of oculars, wherein the optical assembly provides aconvergence angle, the convergence angle being an angle between aleft-eye optical path and a right-eye optical path at the surgical site.In some embodiment, the left-eye camera can comprise: a left-eye turningprism configured to direct light from the surgical site along a left-eyelens path; a left-eye lens assembly configured to receive the directedlight from the left-eye turning prism and to create a left-eye image; aleft-eye image sensor configured to capture the left-eye image andgenerate a left-eye video image. In some embodiment, the right-eyecamera can comprise: a right-eye turning prism configured to directlight from the surgical site along a right-eye lens path; a right-eyelens assembly configured to receive the directed light from theright-eye turning prism and to create a right-eye image; a right-eyeimage sensor configured to capture the right-eye image and generate aright-eye video image. In some embodiments, the left-eye camera and theright-eye camera can be configured to acquire video images of thesurgical site at a convergence point. In some embodiment, a distancefrom the binocular viewing assembly to the convergence point cancomprise a working distance. In some embodiment, the optical assemblycan be configured to provide an adjustable working distance betweenabout 15 cm and about 45 cm. In some embodiment, the left-eye camera cancomprise a left-eye turning prism and the right-eye camera comprises aright-eye turning prism, the left-eye turning prism and the right-eyeturning prism being configured to change their relative orientationsthereby changing the convergence angle to provide the adjustable workingdistance. In some embodiment, the optical assembly can be configured toprovide a substantially constant convergence angle with changing workingdistance. In some embodiment, the left-eye camera and the right-eyecamera can be configured to adjust their relative orientation andposition to provide the substantially constant convergence angle. Insome embodiment, the left-eye camera comprises a left-eye prism assemblyand the right-eye camera comprises a right-eye prism assembly, theleft-eye prism assembly and the right-eye prism assembly can beconfigured to adjust their relative orientation and position to providethe substantially constant convergence angle, wherein other elements ofthe left-eye camera and the other elements of the right-eye cameraremain substantially stationary. In some embodiment, the opticalassembly can be configured to provide a sufficiently narrow convergenceangle to provide stereoscopic imagery through an insertion tube. In someembodiment, the sufficiently narrow convergence angle can be alsosubstantially constant with changes in working distance.

In accordance with another aspect, a medical apparatus can comprise: asurgical retractor; at least one video camera comprising imaging opticsand an optical sensor, said at least one camera disposed on saidsurgical retractor, wherein said imaging optics comprises wafer-scaleoptics. In some embodiments, the medical apparatus can further comprisea stop forward said imaging optics. In some embodiments, said sensor isproximal said imaging optics with said imaging optics between said stopand said sensor, and wherein said stop the most distal optical elementof said camera. In some embodiments, said stop can be just prior to saidwafer optics. In some embodiments, the medical apparatus can furthercomprise a cover plate, said stop disposed between said cover plate andsaid wafer-scale optics, said cover plate comprising sapphire. In someembodiments, the medical apparatus can further comprise a stop withinsaid imaging optics. In some embodiments, said stop can disposed withinsaid wafer-scale optics. In some embodiments, the medical apparatus canfurther comprise a movable optical element within said imaging optics.In some embodiments, said movable optical element can be within saidwafer-scale optics. In some embodiments, said wafer-scale optics caninclude a movable optical element configured to be moved to adjust saidimaging optics. In some embodiments, the medical apparatus can furthercomprise an actuator configured to move said a movable optical elementto adjust said imaging optics. In some embodiments, said imaging opticscan comprise non-wafer-scale lens elements. In some embodiments, saidimaging optics can comprise a negative power non-wafer-scale lenselement. In some embodiments, said negative power non-wafer-scale lenselement can be disposed forward any wafer-scale optics such that saidwafer-scale optics is disposed in an optical path between said negativepower non-wafer-scale lens element and said optical sensor. In someembodiments, said imaging optics can comprise a negative lens group, astop, and a positive lens group arranged in an optical path forward ofsaid optical sensor such that said stop and said positive group aredisposed between said negative lens group and said optical sensor. Insome embodiments, said stop can be between said negative lens group andsaid positive lens group. In some embodiments, said imaging optics canprovide a field of view of at least 70° and up to 125. In someembodiments, said imaging optics can comprise non-wafer-scale optics, astop, and wafer-scale optics. In some embodiments, said non-wafer scaleoptics can comprise negative optical power and said wafer-scale opticscomprise positive optical power. In some embodiments, said imagingoptics can comprise a stack of non-wafer-scale optics lenses havingnegative power, a stop, and positive lens group. In some embodiments,said imaging optics can comprise a front stop, a positive lens, anegative lens, and a plurality of lenses having positive power disposedin an optical path such that the negative lens is between the positivelens and the plurality of lenses having positive power. In someembodiments, said imaging optics can provide a field of view betweenabout 50°-70°. In some embodiments, said imaging optics can comprise afront stop and four wafer-scale optics lenses. In some embodiments, saidimaging optics can comprise no more than four wafer-scale optics lenses.In some embodiments, said imaging optics can provide a field of viewbetween about 50°-70°. In some embodiments, said wafer-scale optics cancomprise a stack of wafer scale optics elements having air gapstherebetween, wherein said air gaps are in fluid communication with eachother. In some embodiments, said wafer scale optics can comprisefiducials on plates that provide stress to counteract bowing. In someembodiments, said fiducials can be interlocking. In some embodiments,said imaging optics can be configured to be disposed laterally withrespect to said optical sensor to provide camera pointing. In someembodiments, the medical apparatus can further comprise actuators tomove said imaging optics laterally with respect to said optical sensorto provide camera pointing. In some embodiments, the medical apparatuscan further comprise actuators to move said optical sensor laterallywith respect to said imaging optics to provide camera pointing.

In accordance with another aspect, a medical apparatus can comprise: atleast one video camera comprising imaging optics and an optical sensor;and a platform configured to be disposed on a surgical retractor, saidat least one video camera disposed on said platform, wherein saidimaging optics comprises wafer-scale optics.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an embodiment of a surgical visualization.

FIGS. 2A-C show an embodiment of a surgical retractor device having anintegrated imaging assembly.

FIG. 3A shows an embodiment of a surgical retractor device having anintegrated imaging assembly.

FIG. 3B shows an embodiment of an imaging assembly containing electricallines and cameras integrated with the retractor blades.

FIG. 3C shows an embodiment of an imaging assembly in which theelectrical lines and cameras are integrated into a flexible cable thatcan be fastened to the retractor frame and retractor blades.

FIG. 4A shows an embodiment of a rotatable stage attached to a retractorframe.

FIG. 4B shows a bottom view of a rotatable stage.

FIG. 4C shows a side view of an embodiment of a camera and prism to bemounted on the inside surface of the rotatable stage.

FIG. 4D shows an enlarged view of a stereo camera pair.

FIG. 5 shows an embodiment comprising a plurality of interchangeableretractor blades for a surgical retractor.

FIG. 6A shows an embodiment of a malleable retractor blade with anintegrated optical sensor.

FIG. 6B shows an embodiment of a retractor blade that is flexible andhas hinges to enable flexure.

FIGS. 6C-D show an embodiment of a rigid articulating retractor blade ina flexed position.

FIGS. 6E-G show an embodiment of a rigid articulating retractor blade inan unflexed position.

FIG. 6H shows an embodiment of a retractor blade stage.

FIG. 7 shows embodiments of the distal end of clip-on flexible cablethat is attached to a retractor blade.

FIG. 8 shows a front surface of a retractor blade or of a flexible cablethat can be attached to a retractor blade.

FIGS. 9A and 9B show embodiments of an aggregator, with one or multipleflexible cables in the rolled configuration.

FIGS. 10A and 10B show embodiments of the aggregator, with one ormultiple flexible cables in the unrolled configuration.

FIG. 11A shows an embodiment of an aggregator, with one or multipleflexible cables with cutouts in the rolled configuration.

FIG. 11B shows an embodiment of an aggregator, with one or multipleflexible cables with cutouts in the unrolled configuration.

FIG. 11C shows an embodiment of an aggregator, with one flexible cablewith cutouts in the unrolled configuration.

FIG. 12A shows an embodiment of a clip-on fastener for fastening theflexible cable to the retractor blade surface.

FIG. 12B shows an embodiment of a hairpin attachment fastener forfastening the flexible cable to the retractor blade surface.

FIG. 12C shows a top view of an embodiment with a dovetail attachmentfastener for fastening the flexible cable to the retractor bladesurface.

FIG. 13A shows the surgical retractor of FIGS. 2A-C with a laser devicepositioned through the opening.

FIG. 13B shows the surgical retractor of FIGS. 2A-C with a needle holderpositioned through the opening.

FIG. 14 shows an example surgical system including an imaging surgicalsystem having an image processing system and cameras associated withsurgical devices.

FIG. 15 shows an example output for display of stereo imagery from aproximal stereo camera overlaid with wide field of view imagery from aproximal wide field of view camera.

FIG. 16 shows the example output for display of FIG. 15 with additionalimagery from distal cameras displayed as well.

FIG. 17 shows an example display incorporating image data from proximaland distal cameras, as in FIG. 16, along with a picture-in-picture viewof imagery acquired by a camera associated with a surgical tool.

FIG. 18 shows an example of using two or more cameras to create amorphed image to provide a central view of a surgical site.

FIG. 19 shows an example configuration of proximal cameras that providea wide field view and a stereo view.

FIG. 20A shows an example configuration of proximal cameras thatmaintain a desired alignment relative to the gravity vector (a directionassociated by the surgeon as opposite to the top of the surgical fieldas viewed from above the patient's body) as well as shows rotating ornot rotating imagery from a camera associated with a surgical tool.

FIG. 20B shows an example configuration of optical sensors mounted onretractor blades to maintain a consistent horizon between a horizon ofacquisition, a horizon of display, and a surgeon horizon.

FIG. 20C illustrates a retractor camera configuration and displaywherein the imagery from the retractor cameras is displayed as tiles onthe display wherein their displayed location corresponds to theirlocations on the retractor and/or fields of view with respect to eachother and/or the retractor.

FIG. 20D illustrates a retractor camera configuration and displaywherein the imagery from the retractor cameras is displayed as tiles onthe display wherein their images are rotated via image processing to bemore consistent.

FIG. 21 shows an example configuration of a proximal stereo camerahaving a first field of view and a second monocular distal camera havinga second field of view.

FIG. 21B illustrates an embodiment of the surgical visualization systemhaving an articulating arm for an imaging system that can be configuredto provide imagery similar to a direct-view surgery microscope.

FIG. 21C illustrates an example surgical viewing system attached to anarticulating arm, the system including one or more cameras mounted on aviewing platform.

FIGS. 21D and 21D-2 illustrates an example surgical viewing system thatincludes an isocenter positioning system attached to the viewingplatform.

FIGS. 21E and 21F illustrate an embodiment of a surgical visualizationsystem having an optical system mounted under the viewing platform.

FIGS. 21G-21K illustrate embodiments of optical assemblies for use in astereoscopic surgical viewing system, such as those illustrated in FIGS.21E-F.

FIGS. 21L-21Q illustrate embodiments of a visualization display withviewing platform attached to a movable arm.

FIGS. 22A and 22B show examples of displaying a composite image bystitching and tiling images from cameras.

FIGS. 23A and 23B show example stitched or tiled displays incorporatingimage data from a plurality of cameras positioned on retractors.

FIG. 24 shows an example graphical user interface that can be used inembodiments of surgical visualization systems.

FIG. 24B shows another example graphical user interface that can be usedin embodiments of surgical visualization systems.

FIGS. 25A-C show an irrigation assembly for cleansing an optical sensor.

FIG. 26 shows a fenestrated ring configured to cleanse optical sensors.

FIG. 27A shows some embodiments of wafer-scale optics for use with asurgical device.

FIG. 27B shows substrates configured to be interlocked to formwafer-scale optics.

FIG. 27C shows a wafer to be diced to provide a plurality of substratesfor forming the wafer-scale optics.

FIG. 28 shows an embodiment of an optical prescription for an imagingmodule having a field of view less than or equal to about 70 degrees.

FIG. 29A shows an embodiment of a wide field-of-view optical assemblywith a buried stop for use with a surgical device.

FIG. 29B shows an example embodiment of an optical assembly comprisingan afocal module coupled to an optical imaging module for use with asurgical device.

FIG. 29C shows an imaging module comprising interchangeable opticalelements configured to change imaging properties of the imaging module.

FIG. 29D shows an example imaging module with optics providing a viewingangle relative to a surgical tool axis.

FIG. 30A shows an embodiment of an imaging stack comprisingnon-wafer-scale optics in combination with wafer-scale optics.

FIG. 30B shows an example embodiment of an imaging stack comprising astop between a negative lens group and a positive lens group, whereinthe positive lens group has a distortion-correcting lens element.

FIGS. 31A and 31B respectively show top and side views of someembodiments of an imaging module comprising an imaging stack, sensorlayer, via layer, and flex layer.

FIGS. 32A and 32B illustrate two embodiments of light guides for usewith imaging modules to provide illumination.

FIGS. 33A and 33B illustrate top and side cross-section views,respectively, of a retractor blade with a light guide and illuminationsource integrated therein.

FIG. 34 is a schematic illustration of one embodiment of a hydraulicactuation system.

FIG. 35A is a schematic illustration of another embodiment of ahydraulic actuation system.

FIG. 35B is a schematic illustration of another embodiment of ahydraulic actuation system.

FIG. 35C is a schematic illustration of another embodiment of ahydraulic actuation system.

FIG. 36A is a schematic illustration of a portion of another embodimentof a hydraulic flow circuit.

FIG. 36B is a schematic illustration of a portion of another embodimentof a hydraulic flow circuit.

FIG. 37 is a schematic illustration of one embodiment of a hydraulicpressure circuit.

FIG. 38 is a schematic illustration of another embodiment of a hydraulicpressure circuit.

FIG. 39 is a schematic illustration of another embodiment of a hydraulicpressure circuit.

FIG. 40 is a schematic illustration of another embodiment of a hydraulicpressure circuit.

FIG. 41 is a schematic illustration of another embodiment of a hydraulicpressure circuit.

FIG. 42 is a schematic illustration of another embodiment of a hydraulicpressure circuit.

FIG. 43 is a schematic illustration of another embodiment of a hydraulicpressure circuit.

FIG. 43A is a schematic illustration of another embodiment of ahydraulic pressure circuit.

FIG. 43B is a schematic illustration of another embodiment of ahydraulic pressure circuit.

FIG. 43C is a schematic illustration of another embodiment of ahydraulic pressure circuit.

FIG. 43D is a schematic illustration of another embodiment of ahydraulic pressure circuit.

FIG. 44A shows a perspective view of one embodiment of a hydraulicmanifold.

FIG. 44B shows an exploded view of the hydraulic manifold of FIG. 44A.

FIG. 44C shows a rear view of the hydraulic manifold of FIG. 44A.

FIG. 44D shows a perspective view of one embodiment of a hydraulicmanifold.

FIG. 44E shows an exploded view of the hydraulic manifold of FIG. 44D.

FIG. 44F shows a rear view of the hydraulic manifold of FIG. 44D.

FIG. 45A shows a perspective cross-section of a hydraulic turbine.

FIG. 45B shows a cross-section of a portion of the hydraulic turbine ofFIG. 45A.

FIG. 45C shows a cross-section of a portion of the hydraulic turbine ofFIG. 45A and a diverted fluid flow path.

FIG. 46 shows one embodiment of an impeller.

FIG. 47A is a schematic illustration of another embodiment of ahydraulic actuation system coupled to a hydraulically actuated surgicaldevice, where the hydraulic actuation system is in a first operatingstate.

FIG. 47B is a schematic illustration of the hydraulic actuation systemof FIG. 47A, where the hydraulic actuation system is in a secondoperating state.

FIG. 47C is a schematic illustration of another embodiment of ahydraulic actuation system coupled to a hydraulically actuated surgicaldevice, where the hydraulic actuation system is in a first operatingstate.

FIG. 47D is a schematic illustration of another embodiment of ahydraulically actuated surgical device.

FIG. 47E is a schematic illustration of an embodiment of a cutting tipof the surgical device embodiment of FIG. 47D.

FIG. 47F is a schematic illustration of another embodiment of a cuttingtip of the surgical device embodiment of FIG. 47D.

FIG. 47G is a schematic illustration of another embodiment of a cuttingtip of the surgical device embodiment of FIG. 47D.

FIG. 48A is a schematic illustration of another embodiment of ahydraulically actuated surgical device in a first position.

FIG. 48B is a schematic illustration of the embodiment of thehydraulically actuated surgical device of FIG. 48A in a second position.

FIG. 49A is a schematic illustration of another embodiment of ahydraulically actuated surgical device in a first position.

FIG. 49B is a schematic illustration of the embodiment of thehydraulically actuated surgical device of FIG. 49A in a second position.

FIG. 50A is a schematic illustration of another embodiment of ahydraulically actuated surgical device in a first position.

FIG. 50B is a schematic illustration of the embodiment of thehydraulically actuated surgical device of FIG. 50A in a second position.

FIG. 51A is a schematic illustration of another embodiment of ahydraulically actuated surgical device in a first position.

FIG. 51B is a schematic illustration of another embodiment of ahydraulically actuated surgical device in a second position.

FIG. 52 is a schematic illustration of another embodiment of a hydraulicactuation system coupled to one or more hydraulically actuated surgicaldevices.

FIG. 53 is a schematic illustration of another embodiment of ahydraulically actuated surgical device.

FIG. 53A is a schematic illustration of an embodiment of a powereddrill.

FIG. 53B is a schematic illustration of an embodiment of poweredscissors.

DETAILED DESCRIPTION

The following description is directed to certain embodiments for thepurposes of describing the innovative aspects of this disclosure.However, a person having ordinary skill in the art will readilyrecognize that the teachings herein can be applied in a multitude ofdifferent ways. The described embodiments may be implemented in anydevice or system that can be configured to provide visualization of asurgical site. Thus, the teachings are not intended to be limited to theembodiments depicted solely in the figures, and described herein, butinstead have wide applicability as will be readily apparent to onehaving ordinary skill in the art.

Surgical Field Visualization

In order to provide for improved visualization of the surgical site, asurgical device can be provided with multiple cameras integratedtherein. For example, the surgical device can be a retractor, and aplurality of cameras may be mounted on or within the retractor. In otherembodiments, the surgical device can be a platform having camerasmounted thereon, but may be separate from any retractor used during thesurgery. Each of the cameras may capture a distinct view of the surgicalsite. Imagery from the plurality of cameras may be integrated together,for example, “stitched” together to form composite mosaic imagery thatmay be displayed disposed over background imagery, or arranged in anarray of tiled images shown individually disposed over backgroundimagery for particular emphasis of a surgical work area. Tiled,individual, or composite imagery can provide the user with a view of thearea of the body on which surgery is being performed. The user canselect the imagery to be displayed and the manner in which it isdisplayed for enhanced utility during surgery. As used herein, the termimagery and images includes video and/or images captured from one ormore cameras. Images from video are often referred to as video images orsimply images. The term images may also refer to still images or snapshots. Video feed or video stream may also be used to describe the videoimages such as video images from a camera.

Such cameras can be of particular use when disposed on surgical devices,such as retractors, which are at least partly disposed within theopening through which the surgery is being performed, so as to providethe user with a perspective of being within or just about within thebody. Retractors, for example, may be used to hold open a region in thebody where surgery is to be performed. This region is formed by makingan incision to provide access to the region or surgical site.Accordingly, various embodiments described herein pertain tonon-percutaneous procedures, for example, non-laparoscopic. Additionallyvarious embodiments described herein pertain to non-endoscopicprocedures. Additionally, various embodiments employ open surgery andcut-down as opposed to percutaneous procedures. Likewise variousembodiments employ larger incisions than would be made for arthroscopy,laparoscopy etc. Various embodiments described herein pertain tominimally invasive surgery (MIS) for spine surgery, all of neurosurgeryand trans-oral approaches to various cancers such as tongue, tonsils,oral and nasal pharynx and anterior skull base (brain). The region mayhave an area of, for example, from 400 to 2500 mm², 1 to 20 cm² or 1 to10 cm² so as to permit the surgeon ready access to the surgical sitesuch that the surgeon can manipulate tools to perform surgery. Forexample, trans-oral surgeries can retract the mandible, maxilla, andeach cheek to provide an approximately 45 by 45 mm working space. Aminimally invasive spine surgery can use a tubular retractor having acircular working space with a diameter of approximately 25 mm. Theretractor contains blades, fingers, or at least one barrier such ase.g., a tube that holds tissue back to maintain open the surgical site.Multiple cameras located on the retractor at locations within thesurgical field or in very close proximity thereto, e.g., within 75 mm ofthe surgical opening, can provide a useful viewpoint for the surgeon.The cameras may for example be located on the blades, fingers, tubularbarrier, or other portion of the retractor close to the surgical fieldor within the patient and the surgical field. The cameras may includepairs of cameras arranged and/or oriented to provide stereo and thus 3Dimaging or single CMOS camera chips with dual optics to provide stereo.The cameras may be located at various locations in relation to surgicaldevices, for example, the cameras can be located proximally and distallyalong or near a retractor, wherein the location of the cameras can beconfigured to facilitate both the progression of surgery and an enhancedview or view selection of an area of interest.

In various embodiments, the retractor maintains a central open regionthat permits the surgeon central access to the surgical field throughthe opening provided by the retractor and cameras and/or stereo camerapairs disposed on the retractor provide views surrounding the surgicalfield. Accordingly, the retractor may comprise a plurality of blades orfingers or other members disposed about an open central region. In someembodiments, the retractor comprises a hollow tube that forms a barrieragainst the tissue surrounding the surgical site. The hollow tube has acentral open region that permits central access to the surgical fieldthrough the tube. Accordingly, in various embodiments, the retractor isdesigned to provide the central open region unobstructed by theretractor to permit tools and other surgical devices to have readyaccess to the surgical site and the central portion of the surgicalsite.

In various embodiments, cameras and/or stereo camera pairs are mountedon the retractor and directed inward toward this central surgical siteto provide a view thereof. Accordingly, in various embodiments thecameras and/or stereo camera pairs surround the central portion or aredisposed about a portion, for example, ¼, ⅓, ½, ⅔, ¾ or more of thesurgical site. The cameras and/or stereo camera pairs may for example bedisposed at 3 or 4 or more points, for example up to 6, 8, 10, or morepoints about the surgical side. For example, the cameras and/or stereocamera pairs may be disposed at positions at 3 o'clock, 6 o'clock, and 9o'clock, or 3 o'clock, 6 o'clock, 9 o'clock, and 12 o'clock as viewedfrom above the surgical site. In other embodiments, the cameras and/orstereo camera pairs may be disposed at 2 o'clock, 6 o'clock, and 10o'clock, or 2 o'clock and 10 o'clock as viewed from above the surgicalsite. These cameras and/or stereo camera pairs may generally face towardthe open central region and thus in some embodiments, at least onecamera and/or stereo camera pair has a field-of-view in which anothercamera and/or stereo camera pair is visible in that field-of-view. Orone or more cameras and/or stereo camera pairs may have a portion of theretractor in their field-of-view in some embodiments.

Accordingly, the cameras and/or stereo camera pairs on a retractor or aplurality of retractors may point in different directions or otherwiseprovide different vantage points. The direction of each camera and/orstereo camera pair may be characterized by its field of view and/oroptical axis. The optical axis may extend outwardly along the center ofthe camera's and/or stereo camera pair's field of view. In someembodiments, the optical axes of two or more cameras and/or stereocamera pairs may be non-parallel and thus not point in exactly the samedirections. In some embodiments, the optical axes of the cameras and/orstereo camera pairs on the retractor may be at an angle greater than 10,20, 30, 40, 50, 60, 70, 80 degree's with respect to each other and maybe substantially orthogonal but less than 20, 30, 40, 50, 60, 70, 80,90, 100 degrees in some embodiments. In some embodiments, the opticalaxes of the cameras and/or stereo camera pairs on the retractor may beat a larger angle such as greater than 90, 100, 110, 120, 130, 140, 150,160, or 170, degree's with respect to each other but less than 100, 110,120, 130, 140, 150, 160, 170 or 180 degrees in some embodiments. In someembodiments, the optical axes of two cameras and/or stereo camera pairsmay be anti-parallel and in various examples may at least be directed inopposite directions. In certain embodiments, these camera and/or stereocamera pairs may be directed at least toward a common central area andmay potentially have field-of-views that at least partially overlap. Insome embodiments, the centerlines of the field-of-views of two camerasor stereo camera pairs may converge at a point or small central area(e.g., less than 500 mm² or less than 100 mm²) within the surgical site.

In some embodiments, the projected angle (as seen from directly abovethe surgical site) between the optical axes of two cameras and/or stereocamera pairs on the retractor may be 0 degrees, however, in variousembodiments it may be at least 5, 15, 30, 45, 60, 75, or 90 degrees butless that 100 degrees such as between 5 and 15, 15 and 30, 30 and 45, 45and 60, 60 and 75, 75 and 90 degrees with respect to one another withthe cameras and/or stereo camera pairs facing inward toward the surgicalsite. More than two cameras and/or stereo camera pairs on the retractormay be oriented such that optical axes of the more than two camerasand/or stereo camera pairs (as seen from directly above the surgicalsite) is at least 5, 15, 30, 45, 60, 75, or 90 degrees such as between 5and 15, 15 and 30, 30 and 45, 45 and 60, 60 and 75, 75 and 90 degreeswith respect to one another.

In some embodiments, the projected angle (as seen from directly abovethe surgical site) between the optical axes of two cameras and/or stereocamera pairs on the retractor may be at least 95, 105, 115, 125, 135,145, 155, 165, or 175 degrees but less that 180 degrees such as between95 and 105, 105 and 115, 115 and 125, 125 and 135, 135 and 145, 145 and155, or 165 and 175 degrees with respect to one another with the camerasor stereo camera pairs facing inward toward the surgical site. More thantwo cameras and/or stereo camera pairs on the retractor may be orientedsuch that optical axes of the more than two cameras or stereo camerapairs (as seen from directly above the surgical site) is at least 95,105, 115, 125, 135, 145, 155, 165, or 175 degrees but less than 180degrees such as between 95 and 105, 105 and 115, 115 and 125, 125 and135, 135 and 145, 145 and 155, 155 and 165, 165 and 175, or 175 and 180degrees with respect to one another.

The cameras and/or stereo camera pairs may be configured to be orientedin such directions. For example, as discussed herein, the cameras may bedisposed on platforms that removably attach to the retractor blades in amanner in which the position and/or orientation of the cameras can bealtered. For such cameras and/or stereo camera pairs, for example, atleast two or more such cameras and/or stereo camera pairs may beconfigured to be at least 5, 15, 30, 45, 60, 75, or 90 degrees but lessthan 100 degrees or 95, 115, 130, 145, 160, 175, degrees but less than180 degrees with respect to one another. The cameras and/or stereocamera pairs can be arranged in many directions to provide asubstantially increased range of views of the surgical site for theuser. Accordingly, the effective field of view provided by the pluralityof cameras and/or stereo camera pairs can be substantially increased.Likewise, although each separate camera and/or stereo camera pair mayhave a small field-of-view, stitching or tiling images from thedifferent cameras and/or stereo camera pairs together may provide alarger field of view. This wide field-of-view within the surgical siteprovides enhanced situational awareness for the surgeon. Embodimentsdescribed herein can be implemented with virtually any retractor system.For example, suitable retractor systems include the ProView MAP system,DePuy SPOTLIGHTR Access System, Metrx X-Tube Retraction System.

In various embodiments, the use of cameras integrated within a retractorcan be applied to neurological, spinal, head and neck, oral, and ENT(ear, nose, and throat) surgeries. Additionally, as described in moredetail below, a separate camera can be integrated with a surgical tool,such as but not limited to a drill, forceps, scissors, Kerrison, bipolarcautery (RF), confocal imager, or laser delivery system.

The cameras may be, for example, Omnivision OV2722 1080P, ⅙ inch. Otherconfigurations are possible. For example, the cameras may includewafer-level optics, conventional optics, molded optics, and combinationsthereof. In various embodiments, the camera may include imaging opticscomprising a negative distal lens group having one or more lenses thatproduce a total optical power for that group that is negative and apositive proximal lens group having one or more lenses that produce atotal optical power for the group that is positive. In certainembodiments, the camera can include an afocal assembly while in otherembodiments the imaging optics are not afocal. In various embodiments,an aperture stop for the imaging optics is between lens elements. Forexample, the aperture stop may be between the negative lens group andthe positive lens group. These lenses may comprise wafer scale optics.In some embodiments, the camera can include wafer-scale optics incombination with non-wafer-scale optics. In some embodiments, the waferscale optical imaging optics comprises layers comprising, e.g., lenselements, separated by spacers that provide air gaps between the layers.The air gaps between the layers may be in communication with each otherand/or air or gas reservoir to reduce the risk of condensation onoptical surfaces.

The cameras may comprise, for example, CCD or CMOS sensor arrays orother types of detector arrays. A frame grabber may be configured tocapture data from the cameras. For example, the frame grabber may be aMatrox Solios eA/XA, 4 input analog frame grabber board. Imageprocessing of the captured images may be undertaken. Such imageprocessing can be performed by, for example, the Matrox Supersight E2with Matrox Supersight SHB-5520 with two Intel Six Core Xeon E5645 2.4GHz processors with DDR3-1333SDRAM. This system can be designed tosupport eight or more camera inputs using two Matrox Solios eA/XA, 4input, analog frame grabber boards. More or less cameras may beemployed. In some implementations, a field programmable gate array(“FPGA”) can be used to capture and/or process imagery received from thecameras. For example, the image processing can be performed by Xilinxseries 7 FPGA boards. Other hardware devices can be used as well,including ASIC, DSP, computer processors, a graphics board, and thelike. The hardware devices can be standalone devices or they can beexpansion cards integrated into a computing system through a localcomputer bus, e.g., a PCI card or PCIe card.

A plurality of illumination sources may be provided to enhance thevisualization provided by the cameras. For example, in some embodiments,each camera can have two LEDs associated with it. The LEDs may bepositioned on opposite sides of the camera, and may be positioned toilluminate the field of view of the camera. The electronic connectionsfor the cameras and/or LEDs can be provided using coaxial cables or flexcables. Flex cables are stronger and more heat resistant than coaxial incomparable sizes. The use of flex cables can allow for a lower profilesystem, since the aspect ratio of width and thickness is significantlyhigher than conventional coaxial or endoscopic approaches. In someembodiments, the illumination element, such as an LED may be directlysoldered onto the flex cable. In various embodiments, both theillumination element and the camera may be permanently affixed to theflex cable by soldering. In some cases an assembly of flex cables can beutilized consisting of a flex cable for CMOS sensor or sensors, a flexcable for one or more LED's, and an EM sensor cable or assembly. Such acombined assembly can be sandwiched as layers in a protective jacket ofsilicone or epoxy resin or Teflon tubing and be attached, affixed, or beoriented with the axis of the retractor blade. Such an assembly mayterminate in an edge connector typically of a male type.

In some embodiments, the cameras can be disposed on one or more surgicaldevices or tools, such as a retractor. The cameras can be positioned andoriented such that one camera is within a field of view of anothercamera. Similarly, cameras can be positioned and oriented such thatother portions of the surgical device can be within a field of view ofthe camera. Labels, fiducials or color markings can be included on thesurgical devices, tools, and/or cameras. The labels or color markingscan be configured to be within a field of view of another camera. Forexample, for a retractor having three blades, each with a camera, eachblade may also include a corresponding label or color marking. Theblades may be labeled, 1, 2, and 3 (or be marked with red, green, orblue) respectively. Likewise, the view from the camera on blade number 1(red) would show the blade with the number 2 (green) marking or both theblade with the number 2 (green) and number 3 (blue) markings. Themarkings should also be visible. Such markings can give the user quickidentification of which camera generated the image (e.g., the camera onblade number 1 (red) in this example). The labels or markings thusprovide increased or enhanced situational awareness to a user oroperator by assisting in understanding the position and/or orientationof a camera associated with respect to a surgical tool or device. Insome embodiments, the markings, fiducials or labels can be used in imageprocessing such as in stitching or tiling processes to form a compositeimage.

In certain embodiments, there are an odd number of cameras and/or stereocamera pair provided with the retractor. In some embodiments, thecameras and/or stereo camera pair can be configured to pointapproximately normal to a retractor axis, and having an odd number ofcameras and/or stereo camera pair views can provide for a central cameraview providing image data for a central portion of a targeted area andsymmetry in the numbers of optical modules views on either side of thecentral camera view providing image information about peripheralregions. In some embodiments, these sensor module views may bestereoscope views. In such embodiments, an additional camera and/orstereo camera pair can be used to provide a view of a targeted feature,such as a view from a surgical tool (e.g., a cutting tool) and thecorresponding area of interest (e.g., area of tissue being cut). In someembodiments, this image of the target features can be displayed in acentral portion of a display. The remaining cameras and/or stereo camerapair on the retractor can provide other visual information, such as toolentrance or egress, background or peripheral visual information.

In some embodiments, an electrical connector can be included in eachblade of a retractor. In some embodiments, each camera module'sconnector may be provided at the distal end of an additional run ofcabling that attaches to a manifold. Such cabling can be color-codedand/or marked so as to identify or indicate stereoscopic view, field ofview differences, etc. The cabling may be plugged into site-specificplugs, which may be unique to the camera type, on a manifold that can,for example, be positioned on the retractor frame near the patient or ona console. In some embodiments, EEPROM tags associated with thedifferent sensors may be used to identify and provide informationrelated to the sensor. In various embodiments, the termination of thecabling is male to facilitate sterilization.

FIG. 1 shows one embodiment of a surgical visualization system. Asillustrated, the system 1 includes a base 3 from which two articulatingarms 5 and 7 extend. The first articulating arm 5 has mounted to itsdistal end a viewing platform 9. The viewing platform may include twooculars 11 and be configured similarly to a standard surgical microscopeviewing platform. In some embodiments, however, unlike a conventionalsurgical microscope or a head mounted display the viewing platform 9 isnot a direct view devices where the surgeon or other user sees directlythrough the platform, e.g., an aperture in the platform. As discussed inmore detail below, the viewing platform 9 may include displays whichreceived signals from cameras which the surgeon or user employs to viewthe surgical site. In some embodiments, cameras can be mounted to theviewing platform 9 and the cameras can be configured to provide imageryof the surgical site. Accordingly, the cameras can be used to provideimagery similar to a conventional surgical microscope. For example, thecameras on the viewing platform can be configured to provide a virtualworking distance, or a distance from the viewing platform to thepatient, that can vary using zooming. The virtual working distance canvary, where the working distance can be at least about 150 mm and/orless than or equal to about 450 mm, at least about 200 mm and/or lessthan or equal to about 400 mm, or at least about 250 mm and/or less thanor equal to about 350 mm. The working distance can be selected and/orchanged by the surgeon. In some embodiments, the cameras mounted on theviewing platform 9 can be used to provide gesture recognition to allow asurgeon to virtually interact with imagery provided by the display usingthe surgeon's hands, a surgical tool, or both, as described in greaterdetail herein. The second articulating arm 5 has mounted to its distalend an input and display device 13. In some embodiments, the input anddisplay device comprises a touchscreen display having various menu andcontrol options available to a user. In some embodiments, thetouchscreen can be configured to receive multi-touch input from tenfingers simultaneously, allowing for a user to interact with virtualobjects on the display. For example, an operator may use the inputdevice 13 to adjust various aspects of the displayed image. In variousembodiments, the surgeon display incorporating a video camera providinga surgical microscope view may be mounted on a free standing articulatedarm. The flat panel display touch screen may be positioned on atilt/rotate device on top of the electronics/fluidics console.

A retractor 15 and surgical tool 17 are both connected to the base 3 byelectrical cables 19. In other embodiments, the retractor 15 andsurgical tool 17 may be in wireless communication with the base 3, forexample via WiFi (IEEE 802.11a/b/g/n), Bluetooth, NFC, WiGig (IEEE802.11ad), etc. As described in more detail below, one or both of theretractor 15 and surgical tool 17 may include one or more camerasconfigured to provide imagery, e.g., image and/or video data. In variousembodiments, video data can be transmitted to a video switcher, cameracontrol unit (CCU), video processor, or image processing modulepositioned, for example, within the base 3. The video switching modulemay then output a display video to the viewing platform 9. The operatormay then view the displayed video through the oculars 11 of the viewingplatform 9. In some embodiments, the binoculars permit 3D viewing of thedisplayed video. As discussed in more detail below, the displayed videoviewed through the viewing platform 9 may comprise a composite videoformed (e.g., stitched or tiled) from two or more of the cameras on theretractor 15 and/or surgical tool 17.

In use, an operator may use the retractor 15 and surgical tool 17 toperform minimally invasive surgery. The operator may view the surgicalsite by virtue of the displayed imagery in the viewing platform 9.Accordingly the viewing platform (surgeon display system) 9 may be usedin a manner similar to a standard surgical microscope although asdiscussed above, the viewing platform need not be a direct view devicewherein the user sees directly through the platform 9 to the surgicalsite via an optical path from the ocular through an aperture at thebottom of the viewing platform 9. Rather in various embodiments, theviewing platform 9 includes a plurality of displays, such as liquidcrystal or light emitting diode displays (e.g., LCD, AMLCD, LED, OLED,etc.) that form an image visible to the user by peering into the ocular.Accordingly, one difference, however, is that the viewing platform 9itself need not necessarily include a microscope objective or a detectoror other image-capturing mechanisms. Rather, the image data is acquiredvia the cameras of the retractor 15 and/or the surgical tool 17. Theimage data can then be processed by a camera control unit, videoprocessor, video switcher or image processor within the base 3 anddisplayed imagery may then be viewable by the operator at the viewingplatform 9 via the display devices, e.g., liquid crystal or LEDdisplays, contained therein.

Imaging Assembly

FIGS. 2A-C show one embodiment a surgical retractor device that includesan integrated imaging assembly. In some embodiments, the imagingassembly includes a plurality of integrated cameras. The retractor 100includes three blades 101, however, more or less may be includeddepending on the design. Each of the blades may be attached to anarticulable arm 103 that allows for the position of the blades to beadjusted during the operation. For example, following a small incision,the three blades 101 can be arranged in a closed position where each arepositioned close to one another. In this closed configuration, the threeblades can be introduced through the incision, and then expanded toprovide for an operating pathway or working space. In other embodiments4, 5, 6, 7, 8 or more blades, fingers, retractor members, or otherbarriers may be employed (or fewer members such as two blades, etc., oreven a single member such as a single lumen of a tubular retractor maybe used). In various embodiments, the surgical area may be at least 400mm², for example, have an opening with an areas between 400 and 2100mm². The working space may be an area centrally located betweenretractor blades (or within the lumen of a tubular retractor) thatallows for surgical tools or other instruments to pass through. Asshown, the retractor does not obstruct the center of the retractor(e.g., array of retractor blades, finger, members, etc., or lumen of atubular retractor) and the open region formed by the retractor andpermits unobstructed access to the center of the surgical site for readyaccess by the surgeon. Each of the blades 101 includes one or moreintegrated cameras, or cameras with combined stereo paths to one sensoror camera module 105. In various embodiments the number of cameramodules and configurations can vary. In the illustrated embodiment, eachcamera module 105 includes a camera 107 and one or more, or twoillumination sources 109 disposed on opposite sides of the camera 107.In various embodiments, the number of illumination sources per cameramodule may vary. In some embodiments, the illumination sources may notbe disposed directly adjacent any particular camera. In someembodiments, the illumination sources can be omitted, and the cameramodule can rely on ambient supplementary or overhead light or directedfrom a light source located elsewhere. In some embodiments, theorientation of an integrated camera 107 may be substantially fixed withrespect to the retractor blade 101 or other surgical tool. In someembodiments, the camera 107 and/or the camera module 105 may beadjustable with respect to the retractor blade 101.

In the illustrated embodiment, the retractor blades 101 aresubstantially rigid. In various embodiments, the retractor blades may bemalleable, and may have a wide range of different structural featuressuch as width, tension, etc. For example, stronger, larger retractorblades may be desired for spinal and trans-oral surgery, while weaker,smaller retractor blades may be desired for neurosurgery. In someembodiments, the retractor can be configured such that different bladescan be arranged as desired.

Each of the camera modules 105 are in electrical communication with theaggregator 104. The aggregator 104 is configured to receive input fromeach of the camera modules 105, and to connect to external componentsvia electrical cable 108. For example, the hub or aggregator 104 mayreceive image data from each of the camera modules 105 and may transmitthe image data to an image processing module (not shown). In theillustrated embodiment, the wiring connecting the camera modules 105with the aggregator 104 is imbedded within the retractor blades 101 andarticulating arms 103 and is not visible. In some embodiments, asdescribed in more detail below, cables connecting the camera modules 105with the aggregator 104 may be adhered (either permanently ornon-permanently, e.g., releasably) to the exterior surface of theretractor 100. In the illustrated embodiment, the hub or aggregator 104is affixed to an upper surface of the retractor 100. The aggregator maybe positioned at any locations relative to the retractor 100, or may bedisconnected from the retractor 100 altogether. The aggregator maycontain camera interface electronics, tracker interface electronics andSERDES to produce a high speed serial cable supporting all cameras inuse. The serial cable extending from the aggregator is preferably maleterminated to facilitate sterilization.

Although the illustrated embodiment shows integrated camera modules 105,in various embodiments the camera modules 105 may be removably attachedto the retractor blades 101. In some embodiments, the camera modules 105can be disposed within pre-positioned receptacles on the retractorblades 101 or other surgical device. In some embodiments, the cameramodules 105 can be disposed at a plurality or range of locations desiredby the user on the retractor blades 101. In various embodiments, theorientation and position of the sensors can be adjusted by the user,e.g., physician, nurse, technician, or other clinician. In someembodiments, for example, the camera may be disposed on a track suchthat the camera can slide up and down the retractor, e.g., retractorblade. The height of the camera or camera within or above the surgicalsite may thereby be adjusted as desired. Other arrangements forlaterally adjusting the position of the camera may be used.Additionally, in various embodiments, the cameras may be configured tohave tip and/or tilt adjustment such that the attitude or orientation ofthe camera may be adjusted. The line of sight or optical axis of thecameras can thereby be adjusted to, for example, be directed moredownward into the surgical site or be directed less into the surgicalsight and more level or angled in different lateral directions. Thecamera modules 105 can include sensors or markers for, e.g.,electromagnetic or optical tracking or use encoders accelerometers,gyroscopes, or inertial measurement units (IMUS) or combinations thereofor any other orientation and/or position sensors, as described in moredetail below. Tracking can provide location and/or orientation of thecameras. The images obtained by the cameras may be stitched together ortiled using image processing techniques to render a composite mosaicimage. Tracking or otherwise knowing the relative locations of thesensor can assist in image processing and display formatting. Trackingposition of cameras can support the touch screen user interface, suchthat a user can select, position and size (zoom) an image array onsurgeon display.

In various embodiments, pairs of cameras together provide informationfor creating a stereo effect or 3-dimensional (3D) image. Pairs ofcameras, for example, may be included on each of the blades 101 of theretractor 100. In certain embodiments, images from separate cameras onseparate blades 101 can be assembled to provide the stereo and threedimensional effect.

As illustrated, the retractor is configured to hold open tissue so as toproduce an open region or cavity centrally located between the blades.Notably, in various embodiments, this open central region isunobstructed by the retractor. In particular, the central portions ofthe open region would be unobstructed by features of the retractor suchthat the surgeon would have clear access to the surgical site. Thesurgeon could thus more freely introduce and utilize his or her tools onlocations within the surgical site. Additionally, this may enable thesurgeon to use tools with both hands without the need to hold anendoscope.

Also as illustrated, the cameras are disposed on the blades of theretractor such that the cameras face inward toward the surgical sitethat would be held open by the retractor blades. The cameras in thisexample would be disposed about the central open region held open by theretractor blades so as to provide views from locations surrounding thesurgical site. The camera thus would face objects within the surgicalsite such as structures on which tools would be used by the surgeon tooperate.

In this particular example, the cameras on two of the blades face eachother such that the leftmost blade and the cameras thereon would in thefield-of-view of the cameras on the rightmost blade and vice versa. Thecameras on the leftmost blade may be anti-parallel to the cameras on theright most blade and have optical axes oriented at an angle, θ, of 180°with respect to each other. The cameras on the remaining blade may bedirected orthogonally to the other two blades and thus have optical axesdirected at an angle, θ, of 90° with respect to each other. Retractorswith cameras can be reaffixed to frame or mounting structure during aprocedure and cameras reorient themselves with respect to relativeposition within an array of cameras through their communication protocolwith the aggregator and video switching unit.

In some embodiments, the field-of-views of the different cameras, andhence the images produced by the different cameras, may overlap. Imageprocessing may be employed to yield increased resolution at the regionsof overlap. Likewise, the number of sensors used may be increased toprovide increased field-of-view and/or resolution. Likewise, cameraswith overlapping images can be electronically magnified thereby makingtheir images adjacent rather than overlapping.

FIG. 3A illustrates another embodiment of a surgical retractor device6014 having an integrated imaging assembly shown with a different wiringarrangement than shown in FIG. 2A. In particular, much of the wiring isexposed as opposed to being buried in the retractor. As illustrated, theimaging assembly 6013 comprises a hub or aggregator 104 and one or moreimaging subassemblies 6011. The imaging assembly contains three imagingsubassemblies 6011 although more or less may be included. Asillustrated, in some embodiments, the imaging subassemblies 6011 can beintegrated within the surgical retractor. Also as illustrated, theimaging subassemblies 6011 can contain electrical signal lines 106 andoptics and/or sensor elements. In various embodiments, the electricalsignal lines 106 can comprise cables or lines that are exposed andvisible as opposed to completely imbedded within the retractor. In otherembodiments, different portions of the lines may be buried or embeddedwithin the portions of the retractor and thus not exposed. Theelectrical signal lines 106 can connect at a first end to an aggregator104 and at a second end to a camera 6012. The camera can contain optics(e.g., imaging lenses) and a sensor element (e.g. a two-dimensionaldetector array such as a CMOS or CCD 2D-array). The electrical signallines 106 are combined together within the aggregator 104 to a commonbus line. The image processing system cable 108 electrically connects tothe aggregator 104, e.g., to the common bus line, at a first end and animage processing unit at a second end (not shown). The imagingsubassemblies 6011 can be electrically connected to the aggregator 104through a male/female connection. In various embodiments, the first endof the electrical signal lines 106 comprises a male connection, whilethe receiving aggregator 104 connection comprises a female connection.The aggregator 104 (having, for example, female connectors) can be adisposable unit. The imaging subassemblies 6011 (having, for example,male connectors) can be reusable and can be cleaned and/or sterilized.The line 108 can also be terminated by a male connector that plugs intoa female connector on the aggregator 104. Similarly, the long cable 108can be reusable and sterilizable.

FIG. 3B illustrates another embodiment of an imaging assembly in whichthe electrical lines or cable, for example, flex cable, and cameras areintegrated with the retractor blades or connect to a blade havingelectrical pathways therein or thereon. In certain embodiments, theretractor blades 7010 can contain varying optics, sensors (e.g., 2Ddetector arrays), and lighting (e.g., light sources such as LEDs,superluminescent diodes, supercontinuum light sources, or xenon).Additionally, in certain embodiments the retractor blades 7010 can havevarying widths, lengths, and strengths. As illustrate, the retractorblades 7010 can be removably attached to the retractor frame 7002. Inthe embodiment shown, the retractor blade includes a protruding railmember that slidably fits into a track on an arm of the retractor. Thusthe surgeon is able use different retractor blades with differentcomponents, sizes, and strengths thereby permitting the surgeon or userthe flexibility to provide the suitable retractor blades or optics for aparticular medical procedure.

FIG. 3C illustrates an embodiment of an imaging assembly in which theelectrical lines and cameras are integrated into a flexible cable thatcan be readily fastened to the retractor frame and retractor blades. Incertain embodiments, the flexible cable 7001 can be affixed to theretractor frame 7002 and the retractor blades 7003 in a manner to beeasily affixed and removed. In some embodiments, the flexible cable 7001can have a distal end 7005. The distal end 7005 of the flexible cable7001 can contain optics, sensors, or lighting and combinations thereof.In various embodiments, the flexible cable 7001 can contain a camera7004. In certain embodiments, the flexible cable 7001 and the aggregator7007 can be clipped on to the retractor blades 7003 and retractor frame7002, respectively. In one embodiment, the distal end 7005 of theflexible cable 7001 can contain a fastener member 7008 for fastening andunfastening the flexible cable to the retractor frame and/or theretractor blades as shown in FIG. 3C. The fastener member 7008 caninclude, for example, a clip, a snap, a strap, a screw, a bolt, a nut,or any combination of these as well as any other method that canfacilitate convenient attachment. For example, attachment can beaccomplished in under one minute possibly less than 20, 10, 5, 3, or 2seconds per fastener and may be accomplished in more or less than asecond or ½ or ¼ second per arm such that attachment can occur, forexample, just prior to and in preparation for surgery. In someembodiments, aggregator 7007 can be permanently attached to theretractor frame 7002. In such embodiments, the flexible cables 7001 canconnect to the aggregator and be removably fastened to the retractorframe 7002 and/or the retractor blades 7003 with a fastening member7008. In some embodiments, additional fasteners, not shown, may belocated elsewhere, for example, to attach the flex cable 7001 to theretractor arms or other portions of the retractor.

The retractor blades 7003 and the flexible cable 7001 can extend intothe interior of the body cavity or surgical field opening when theretractor is in use. For example, the retractor blades can be used tohold open the surgical field. The cameras or sensors (e.g., CMOS or CCDdetector arrays) integrated into the retractor blades or integrated intothe flexible cable and clipped onto the retractor blades can produceimages of the surgical field within the body cavity.

Rotatable Stage or Frame

During surgery it may be desirable to rotate a plurality or array ofcameras and/or stereo camera pairs as a group together with respect tothe surgical field, patient, or retractor. For example, a surgeon'spositioning relative to the surgical field can vary for differentprocedures and different surgeons or the positioning of the retractorblades may be set at an angle that does not provide for optimal imagingor image processing. Therefore, it may be useful to rotate or otherwisechange the positioning of the plurality of cameras and/or stereo camerapairs without changing the retractor blades positioning.

It can be beneficial, for example, that the optics and sensor be in aposition to produce an image of the surgical field having vertical andhorizontal directions the same or substantially the same as the verticaland horizontal directions that the surgeon associates for the surgicalfield. If the cameras are not positioned correctly, the image of thesurgical field may be rotated on the display such that vertical andhorizontal directions on the display do not correspond to vertical andhorizontal directions that the surgeon associates with the surgicalfield as oriented for the surgical procedure. Incorrect positioning orexcessive rotation of the surgical field with respect to the verticaland horizontal directions on the display can decouple hand-eyecoordination.

Accordingly, various embodiments may include a rotatable support for thearray of cameras and/or stereo camera pairs that can rotate or move withrespect to the retractor blades. FIG. 4A illustrates an embodiment of arotatable stage attached to a retractor frame. In some embodiments, theplurality of cameras can be mounted on a rotatable stage 8005 and therotatable stage can be coupled to the retractor frame. In someembodiments, the rotatable stage 8005 is fixed to the retractor framethrough an attachment post 8006. The rotatable ring can have a single ormultiple attachment posts 8006 supporting the rotatable stage on theretractor frame. The rotatable stage 8005 can move, for example, rotate,with respect to the retractor blades to permit the image formed by thecamera to rotate on the display. Accordingly, the image of the surgicalfield can be rotated such that the directions on the surgical field thatthe surgeon or operator associate with vertical and horizontalcorrespond to the vertical and horizontal directions of the display.

In certain embodiments, the image visible on the display can be rotatedusing image processing. For example, the surgeon or user can input tothe image processor the amount of rotation that is desired. For example,the user can simply rotate the displayed composite image such as astitched or tiled image as desired. Accordingly, the user can rotate thedisplayed image until such the directions on the surgical field that thesurgeon or operator associates with vertical and horizontal correspondto the vertical and horizontal directions of the display. FIG. 4A showsa plurality of cameras located toward the distal end of the retractor.As illustrated, these cameras are not disposed on a rotating support. Invarious embodiments, image processing may be employed to rotate theimage such as the composite (e.g., stitched or tiled) image formed bythese cameras instead of using a rotating support.

When stereo camera pairs are disposed on the retractor about thesurgical site, having a rotating support for the plurality of stereocamera pairs may, however, be useful. FIG. 4B, which shows a bottom viewof a rotatable stage, illustrates the usefulness of having such arotating support for certain embodiments having an array of 3D cameras.In the embodiment illustrated in FIG. 4B, four stereo camera pairs areshown disposed about an annular shaped support. The stereo camera pairsin this example are disposed at 3 o'clock, 6 o'clock, 9 o'clock and 12o'clock. To provide consistent 3D imaging among the cameras, althoughthe cameras at the 6 o'clock and 12 o'clock positions are arranged alonga radial of the annular shaped support, the cameras at the 3 o'clock and9 o'clock positions are arranged along a tangential direction of theannular shaped support. Such an arrangement of the cameras in thevarious stereo camera pairs provides for consistent 3D imaging as theleft and right camera in the stereo camera pairs will be orientedgenerally along the same direction, e.g., parallel to the x-axis in thisexample. This direction may be oriented, for example, along thehorizontal direction of the surgical field. In various embodiments, eachof the stereo cameras may share a common horizon, even if displayed as astereo circle composed of right and left eye images. If these camerapairs were mounted on fixed retractor blades, in contrast to therotatable support, and if the retractor were rotated in the surgicalfield, the direction along which the left and right cameras are alignedmay not necessarily be parallel with the direction that the surgeonassociates with either horizontal (or vertical). Accordingly, rotationto provide that the left and right cameras are aligned along a lineparallel to the direction the surgeon associates with horizontal (orvertical) may be beneficial. Horizontal alignment in particular may bebeneficial for ergonomic considerations. While an operating roommicroscope can be positioned in an oblique position, looking through it,and holding one's head in that position for hours, as is sometimesnecessary in neurosurgery can be painful for a surgeon. In embodimentsin which the retractor and the display are decoupled, ergonomic viewingis possible even if the surgical access is difficult.

In an alternative arrangement, the stereo camera pairs are not used atthe 3 o'clock and 9 o'clock positions and instead monocular cameras areemployed. The stereo cameras at the 6 o'clock and 12 o'clock positionsas well as the mono cameras at 3 o'clock and 9 o'clock positions neednot be mounted on a rotating support. The mono cameras need not rotatebut the 6 and 12 o'clock cameras should ideally be configured ‘in plane’or their R and L camera views should be parallel with the R and Ldisplay views.

Also, in some embodiments stereo proximal camera pairs are mount on arotating supports while distal mono cameras are not mounted on arotating support in a configuration such as shown in FIG. 4A.

Additionally, in some embodiments, some proximal camera modules includestereo camera pairs and some proximal camera modules include only monocameras and not stereo camera pairs and a rotating stage need not beused in such a case for the proximal cameras. For example, stereo camerapairs can be positioned at the 6 o'clock and 12 o'clock positions andnon-stereo camera were at the 3 o'clock and 9 o'clock positions.Accordingly, in various such embodiments, the stereo camera pair(s)together may be positioned halfway between the other non-stereo camerasand/or the non-stereo cameras may be positioned halfway between stereocamera pair(s). Additionally, in various embodiments, stereo camerapairs are on directly opposite sides of the retractor at locations 180°with respect to each other, such as for example at 3 o'clock and 9o'clock.

However, as shown in FIG. 4A, in some embodiments a rotating support8005 is employed. The rotatable stage 8005 can include a ring comprisingconcentric inner and outer rings. In certain embodiments, the inner ring8007 can move relative to the fixed outer ring 8008. The fixed outerring 8008 can be attached to the retractor frame at one or multiplepoints of attachment. The inner ring can be rotatably coupled to fitwithin or on top of the outer ring. The rotatable stage can supportcameras, flexible cables, or other components including but not limitedto 3D cameras, LEDs, tracking, cleaning, temperature control, heating ortherapeutic delivery systems.

The rotatable stage can include a ring that is rotated by manualmovement, motorized movement, or using other actuators. In someembodiments, the rotatable stage ring can contain a bearing surfacebetween the inner ring and outer ring which allows movement of therotatable ring. The bearing surface can include a plain bearing, a ballbearing, roller bearing, or any other bearing surface. Additionally, inother embodiments the rotatable stage can contain methods of couplingthat allow for translational movement between the inner ring and theouter ring so that the rings can move vertically relative to oneanother. In some embodiments, the ability to rotate the rotatable stage,ring, or alternative platform, allows the stereo image acquisitionhorizon to remain horizontal. Further, the rotatable stage can includeencoders or other tracking devices as for example those described hereinto detect movement of the ring and the placement of the cameras.

As illustrated in FIG. 4B, the stereo camera pairs can be directeddownward from the rotating ring as well as inward toward an axis ofrotation of the ring and a central open region established by theretractor. Despite being downward directed, in some embodiments, thecameras on the rotatable stage are also in different orientations withrespect to each other (e.g. rotated differently) as discussed above. Anaxis of rotation is defined by the line bisecting the optical axis ofthe left and right cameras of the first stereo camera pair. In someembodiments, the third stereo camera pair 8011 is rotated 180 degreesrelative to the first stereo camera pair 8009. In some embodiments, thesecond stereo camera pair 8010 can be rotated 180 degrees relative tothe fourth stereo camera pair 8012. The second stereo camera pair 8010and the fourth stereo camera pair 8012 can be rotated plus or minus 90degrees from the first stereo camera pair 8009. The differentconfigurations of the cameras on the rotatable stage can be advantageousto maintain consistency among the camera images of the different stereocameras on the rotatable stage. A R and L camera view can be rotated 180degrees but if so the eye views must be reversed electronically, so thatupside down R is now L, etc.

In certain embodiments, one or more of the cameras on the rotatablestage can be disposed on the inner surface 8016, as opposed to thebottom surface, of the rotatable stage. A prism or other reflector maybe included to redirect the field-of-view from the camera. For example,the camera can be coupled to a prism, similar to the prisms used in thecylindrical retractor shown in FIG. 21 and discussed herein, to allowthe camera on the inner surface of the rotatable stage to be directed ina downward direction into the surgical field. FIG. 4C illustrates anembodiment of a side view of the camera and prism mounted on the insidesurface of the rotatable stage. The camera 8015 is disposed on theinside surface 8016 of the rotatable stage, which has a prism 8014attached thereto and in the optical path of the camera. In particular,the prism has a reflective surface that is disposed in the optical pathbetween the front of the camera or stereo cameras pair and the surgicalfield. This reflective surface may be oriented at an angle with respectto the retractor to direct the optical path at an angle of between 15 to75 degrees, e.g., 45° with respect to the rotation axis of the ring. Thereflective surface of the prism, for example, may be oriented at anangle of 15 to 30 degrees, e.g., about 22.5° with respect to the axis ofrotation of the ring so as to redirect light at an angle of betweenabout 15 and 75 degrees, e.g., about 45°. Such a prism may reduce theprofile of structures extending into the otherwise open region providedby the retractor so as to maintain a substantially unobstructed openingfor the surgeon to access the surgical site. FIG. 4D shows an enlargedview of a stereo camera pair, with intersecting optical axes. In someembodiments a prism having two reflecting surfaces is employed toredirect the optical path.

Retractor Blades

As discussed above, FIG. 3B illustrates an embodiment of an imagingassembly in which the electrical lines and cameras are integrated withdetachable retractor blades. In certain embodiments, the retractorblades 7010 can contain varying optics, sensors, and lighting.Additionally, in certain embodiments the retractor blades 7010 can havevarying widths, lengths, and strengths. As illustrated, the retractorblades 7010 can be removably attached to the retractor frame 7002 via anattachment system 7011 (e.g., a rail or strip that fits into a track onthe retractor arm). The ability to vary the components, sizes, andstrengths of the retractor blades allows the surgeon or user the freedomto use various retractor blades or various optics that are appropriatefor a particular medical procedure.

FIG. 5, for example, illustrates an embodiment comprising a plurality ofinterchangeable retractor blades for a surgical retractor. Theinterchangeable retractor blades can contain various combinations ofcameras, lighting sources, sensors, imaging optics, EM tracker sensorand/or other components such as discussed herein. The retractor blademay also have none of these components and may be employed primarily formechanical purpose such as to hold back tissue in an incision at theperiphery. Accordingly, the retractor blades can be of varying widthsand strengths. Thus, although in some embodiments, the retractor bladesare permanently attached to the retractor frame or frame (they aretypically called frames), in some embodiments, the retractor blades 7010can be removably attached depending on the desired use or imagingrequired. Such retractor blades can be interchanged to achieve thedesired type of retractor blade depending on the procedure to beperformed. For example, the retractor blades for spinal or trans-oralprocedures can be larger and stronger because of the higher forcerequirement, while retractor blades for neurosurgery procedures can beweaker and smaller. A surgeon may also switch out retractor bladesduring a procedure after commencement thereof. In some embodiments, theretractor blades can have aspiration channels or hold aspirators toremove blood and saline or other liquid. Such aspiration channels can beconnected by fluidic lines such as lines in the flex cable to a pump orother vacuum source tubing can also send warmed air towards the camerasto prevent fogging.

Further, the retractor blades can also be flexible. FIGS. 6A and 6Billustrate example embodiments of retractor blades that are flexible.FIG. 6A illustrates an embodiment of the pre-flexed retractor blade. Insome embodiments, the retractor blade can be made of malleable orflexible material that allows for movement of the retractor blade eitherprior to use or during use, e.g., during a surgical procedure. Theretractor blades of the retractor can be made of a flexible or malleablematerial which allows for the bending or movement to adjust the shape ofthe retractor blades but still produce the required stiffness forretractor purposes. In some embodiments, the retractor blades can bemade of pre-flexed Nitinol retractor blades and may include one or twocables to straighten out the flex. In certain embodiments, the cable(s)is located on the surface away from the patient. The retractor blade canbe allowed to bend or move and this movement can be helpful in theimaging of the surgical field and/or assisting in retraction of thetissue in the surgical area. The flexible retractor blades can be fixedto the retractor unit such as described herein for fixed retractorblades herein. In other embodiments, the flexible retractor blades areclipped onto the retractor unit such as described herein for clip-onretractor blades. In some embodiments, bending or movement of theretractor blade can be controlled remotely, for example by electronicremote control. In some embodiments, the retractor blade may be tiltedby at least 10 degrees, 20 degrees, 30 degrees, 40 degrees, or more andless than 90 degrees, less than 80 degrees, less than 70 degrees, lessthan 60 degrees, or less than 50 degrees via the remote control. In someembodiments, remote control can be provided via a graphic interface.

With continued reference to FIG. 6A, the malleable retractor assembly350 includes a retractor blade 351 with a push-pull wire 353 attached tothe surface of the retractor blade 351. The wire 353 can rest in a lowfriction cable guide to enable flexure of the blade 351. An integratedcamera 355 and electromagnetic tracker sensing coil 357 are disposedwithin the retractor blade 351. The flexure mechanism can be similar tothat used for steerable catheters for interventional procedures. In use,the operator may manipulate the individual retractor blade 351 by use ofa small handle 359 that can pull the wire 353 by use of a pulley orother mechanism. For example, the wire 353 can be made of Nitinol, andmay be attached to a pulley such that upon rotation of the handle 359,the retractor blade 351 flexes or extends. In some embodiments, theblade 351 always curves outwards away from the surgical site so as tocreate a working space for the operator. A set screw (cable clamp) 361can be employed to fix the axial position of the wire 353. The set screw361 can be loosened as desired to axially move the wire 353, therebyadjusting the flexure of the retractor blade 351. In other embodiments,two or more wires may be employed and used in conjunction to adjust theflexure of the blade. Also, in other embodiments, the flexible retractorblades can comprise front and rear cables or elastic bands as describedmore fully below.

In various embodiments, the surface of the blade 351 can be coated withPTFE to reduce tissue friction and sticking. In some embodiments, theblade 351 can be coated with a thin layer of elastomeric material or aninflatable balloon to normalize the pressure per area across the entireregion of contact with the body. In other embodiments, two wires may beused, one having a distal attachment, the other with a more proximalattachment. For example, the wires may run along the outer curvature ofa pre-flexed Nitinol retractor blade. Axially moving one or both wiresenables incremental controllable flexure of the blade.

FIG. 6B illustrates an embodiment of a flexible retractor blade with oneor more joints. As illustrated, in some embodiments, the flexibleretractor blades 9003 can be bent at a joint 9004 or multiple jointswithin the flexible retractor blade 9003. Front and rear elastic bands9001, 9002 can be actuated at a proximal end 9005 by one or more motor,piezo, hydraulic actuator, linear actuator, or rotary actuator, or othertype of actuator. The flexible retractor blades can have cables ortendon actuation with or without the Nitinol pre-flex retractor blade bythe use of, for example, pull-pull cables or lines.

Although FIG. 6B illustrates an embodiment of a flexible retractor bladewith front and rear elastic bands 9001, 9002, in certain embodiments,the front and rear elastic bands of a flexible retractor blade 9003 maycomprise an extensor and a flexor cable. The flexor and extensor cablescan allow for a greater range of motion, bi-directional flexing, and anS-shape with the two joints flexing in opposite directions. In variousembodiments, the force of the tissue being retracted causes theretractor blades to return to an unflexed position once tension on thecable or band is released.

FIGS. 6C-6G illustrate an embodiment of a retractor blade having rigidplates or segments connected by discrete joints. In the illustratedembodiment, retractor blade 9050 includes three plates 9052, 9054, and9056. These plates connect to one another at joints 9058 and 9060. Theplates and joints can be manipulated to place the retractor blade in aflexed configuration (as in FIGS. 6C and 6D) or in an unflexedconfiguration (as in FIGS. 6E-6G). Internal cables (not shown) mayextend within the plates 9052, 9054, and 9056, and be attached to pinionkeys 9062 and 9064. These pinion keys 9062 and 9064 can be rotated topull these internal cables. For example, pinion key 9062 may be rotatedto pull a first internal cable that extends to the middle plate 9054.Pinion key 9064 may likewise be rotated to pull a second internal cablethat extends to the outermost plate 9056. Pulling these respectivecables causes the plates to rotate with respect to one another aboutjoints 9058 and 9060. Specifically, rotating pinion key 9062 exerts apulling force on the first internal cable, which causes the middle plate9054 to rotate about joint 9058. Similarly, rotating pinion key 9064exerts a pulling force on the second internal cable, which causes theoutermost plate 9056 to rotate about joint 9060. Ratchets 9066 and 9068operate to restrain the rotary position of the pinion keys 9062 and9064. Depressing ratchet 9066 releases pinion key 9062, which releasestension on the internal cable, thereby permitting the middle plate 9054to rotate back to a position substantially parallel to the innermostplate 9052. Similarly, depressing ratchet 9068 releases pinion key 9064,which releases tension on the second internal cable, thereby permittingthe outermost plate 9056 to rotate back to a position substantiallyparallel to the middle plate 9054. As described elsewhere herein, theretractor blade 9050 may include a camera 9070 thereon. In theillustrated embodiment, the camera 9070 is positioned on the uppersurface of the outermost plate 9056. Cone 9072 illustrates the field ofview of the camera 9070.

In use, retractor blade 9050 may be introduced into an incision, withits upper surface facing a working space, and its lower surface facingthe surface of the tissue to be retracted. Articulating the retractorblade 9050 (for example by rotating pinion keys 9062 and 9064) causesthe plates 9052, 9054, and 9056 to exert pressure on the tissue, therebyincreasing the size of the working area. In some embodiments,articulation of the retractor blade 9050 can be performedelectronically, hydraulically, or by other methods. In some embodiments,the articulation can be controlled remotely. Camera 9070 can bepositioned with respect to outermost plate 9056 such that its field ofview (represented by cone 9072) is directed towards a site of interestwithin the body. As described elsewhere herein, a plurality of suchretractor blades having a plurality of cameras can be used inconjunction to provide for improved visualization of the surgical site.By controlling the position of outermost plate 9056 (for example, bycontrolling the articulation of retractor blade 9050), the position andorientation of the camera 9070 can be controlled. As noted above, thiscontrol may be electronic, hydraulic, or otherwise, and may be performedremotely. In some embodiments, articulation or other movement of theretractor blades can be controlled via a graphic user interface, such asprovided by a touchscreen, via voice command, etc. Throughout anoperation, the position of the camera 9070 may be controlled (e.g., bymanually adjusting the retractor blades, by remote electronic control,or other means) to provide a desired field of view. In some embodiments,bending or movement of the retractor blade can be controlled remotely,for example by electronic remote control. In some embodiments, theretractor blade may be tilted backward or forward by at least 10degrees, 20 degrees, 30 degrees, 40 degrees, or more and less than 90degrees, less than 80 degrees, less than 70 degrees, less than 60degrees, or less than 50 degrees via the remote control. In someembodiments, remote control can be provided via a graphic interface.

In some embodiments, the retractor blades, finger, member, etc. can behydraulically manipulated to control the movement of the retractorblades. The retractor can have one or more hydraulic members that arehydraulically actuated to displace tissue or apply pressure. Theactuator(s) may comprise one or more linear and/or rotary actuators. Alinear actuator may comprise bellows, rolling edge diaphragms,piston-cylinders with hydrostatic bearings, other linear actuators orother actuators including those disclosed herein, known in the art, oryet to be devised. A rotary actuator may comprise displacement typehydraulic motors, vane motors, gerotors, Bourdon tubes, or other rotaryactuators known in the art. In some embodiments, a force feedback hapticinterface could be used to provide tactile feedback to the user. In someembodiments, a dual bellows actuator's hydraulic source could supplysubstantially constant hydraulic force to the hydraulic members. In someembodiments, linear motors could drive master piston-cylinders withhydrostatic bearings for each axis. Other configurations are possible.

In some embodiments, the hydraulic member can be the retractor blades,fingers, members, etc. In some embodiments, the retractor can have amechanical extension to move tissue extending from the retractor or theretractor blade. In such embodiments, the hydraulic member can actuatethe mechanical extension to move tissue, apply pressure, and/or otherfunctions that might be necessary for the surgical preparation orprocedure. In some embodiments, actuation of the retractor blades can becontrolled remotely.

In some embodiments, the retractor or retractor blade, finger, member,etc. can contain a positionable tool holder. In certain embodiments, thepositionable tool holder can be attached to a retractor blade. In someembodiments, the positionable tool holder can be connected to theretractor base. The positionable tool holder when connected to theretractor base can be out of the field of view of the cameras and doesnot obstruct the images produced. Additionally, such placement on theretractor unit leaves the surgical field clear and allows the surgeonmore room to operate. One example is the inclusion of a large boresuction cannula to remove blood and saline mixture, which is coupled tothe positionable tool holder. Another example involves coupling asupplementary light source, such as a fiber optic cable, to thepositionable tool holder.

As discussed above, in some embodiments, the flexible retractor blades,finger, member, etc. can vary in sizes and toughness or durability toaccommodate certain surgical or medical procedures. For example, theretractor blades for spinal or trans-oral procedures can be larger andstronger because of the higher force requirement, while retractor bladesfor neurosurgery procedures can be weaker and smaller. Additionally, theconfigurations for achieving and methods of using the flexiblecharacteristics of the retractor blades can vary such as describedherein. Further, the flexible retractor blades can have the variousoptics, sensors, or other lighting, tracking, or imaging components thatcan be integrated into the retractor blades such as described herein.

In certain embodiments, the flexible retractor blades can containcameras and LEDs, cameras only, LEDs only, or any other combination ofcomponents herein described. As discussed above, in some embodiments,the flexible retractor blade can contain one or more components thatallow for tracking of the location, orientation, or registration of theattached cameras or combinations thereof. The reconfigurable shape ofthe retractor blade can makes the use of the tracking particularlyuseful for touch screen user interface. Accordingly, the flexibleretractor blades can incorporate various methods of tracking including:EM trackers, optical tracking, inertial measurement units (IMUs),encoders, and other methods including but not limited to those describedherein. The flexible retractor blades can, for example, includeencoders, inertial measurement units, such as Hall Effect encoders, todetect the change in position of the flexible retractor blades. In someembodiments, the encoder can supply the user with information such as ameasurement of degrees of movement of the flexible retractor blade. Theencoders can also provide information to track the location of theretractor blade and for example, the camera(s) located thereon.

Further, in some embodiments the flexible retractor blade may be usedfor mechanical purposes only, such as retraction of tissue, and does notcontain any camera, sensors, trackers, or light source components. Forexample, in some embodiments the flexible nature of the retractor bladecan be used to move tissue out of the way. In other embodiments, theflexible nature of the retractor blade can be used to redirect thepoint-of-view of the camera on the flexible retractor blade.Additionally, in other embodiments, the flexible nature of the retractorblade can be used to both move tissue out of the way and redirect thepoint-of-view of the camera on the flexible retractor blade. In otherembodiments, the flexible retractor blades can have aspiration channelsor hold aspirators to remove blood and saline or other liquid. Suchaspiration channels can be connected by fluidic lines in the flex cableto a pump or other vacuum source.

Multiple flexible or malleable retractor blades can close on a central,axial bullet-tip rod to be used as a dilator and introducer like MetrX.In some embodiments, a dedicated tool may be used to open and close theblades closed on a central, axial rod.

In some embodiments, the retractor blade as illustrated and describedwith reference to FIGS. 6C-6G and additional retractor blade embodimentsdescribed herein can be assembled onto a retractor blade stage asillustrated in FIG. 6H. The retractor blade stage 9080 can contain aretractor blade stage ring 9081 and a gear ring 9082. In someembodiments, the retractor blade stage ring 9081 can be fixed and shapedto receive the gear ring 9082. In some embodiments, the outer surface ofthe retractor blade stage ring 9081 can be shaped in an ‘L’ shape andthe gear ring 9082 can sit within the recess of the ‘L’. The outersurface of the gear ring 9082 contains teeth and faces the outside ofthe ring.

In some embodiments, the retractor stage ring 9080 can be used to effecta radial movement of the retractor blades. One or more clamps 9083 canbe attached to portions of the retractor blade stage ring 9081 and thegear ring 9082. The clamp 9083 can have a pinion 9084, a stem 9085, anda stem ratchet 9086. The stem 9085 can have an innermost end attached tothe proximal end 9052 of the retractor blade 9050. The stem 9085 canhave an outermost end that can pass through the clamp 9083. In someembodiments, the stem 9085 can move horizontally through the clamp toeffect a radial movement of the retractor blade. The clamp 9083 can havea stem ratchet 9086 to engage the teeth of the stem 9085 therebyoperating to restrain the radial position of the stem 9085. For example,depressing the stem ratchet 9084 releases the stem 9085 by disengagingthe ratchet 9086 from the teeth of the stem, which releases tension onthe stem 9085, thereby permitting the stem 9085 to move horizontallythrough the clamp 9083. The retractor blades can be positioned insidethe retractor blade stage 9080 at different distances depending on thepositioning of the stem 9085. Additionally, in some embodiments, theretractor stage ring 9081 can be used to effect a radial movement of aplurality of retractor blades.

Additionally, in some embodiments, the retractor stage ring 9081 can beused to effect a rotational movement of a plurality of the retractorblades. In some embodiments, the pinion 9084 can be verticallypositioned within the clamp 9083. The pinion 9084 can have a distal endextending into the clamp and a proximal end that protrudes from the topsurface of the clamp. The distal end of the pinion 9084 can have teeththat can engage the teeth on the gear ring 9082. When the pinion isrotated, the teeth on the pinion 9084 engage with the teeth on the gearring 9082 and the resulting torque causes the clamp 9083 to rotate onthe retractor blade stage 9080. For example, turning the pinion 9084causes the clamp 9083 to rotate on the ring 9081, 9082, the stem 9085that runs through the clamp and connects to the retractor blade rotateswith the clamp, and therefore the rotational position of the retractorblade 9050 can be changed by the rotational movement of the clamp 9083on the ring. Alternatively, in some embodiments, the retractor stagering 9081 can be used to allow the rotation of each blade individually.

Additionally, in some embodiments, the retractor blade stage 9080 canallow for the flexure of the retractor blades. The retractor blades canbe flex-finger retractors as described herein with reference to 6C-6G.Alternatively, the flexure retractor blades can be achieved in retractorblades without joints by a tilting of the retractor blade at an anglewith respect to the longitudinal axis of the retractor blade stage. Thetilt of the retractor blades can assist in the manipulation of tissue orholding of tissue within the surgical area as well as possibly tilt thecamera orientation.

In some embodiments, the retractor blade stage can allow for rotational,radial, and/or flexure movement of the retractor blades to assist in theappropriate positioning of the retractor blades and this movement can behelpful in the imaging of the surgical field and/or assisting inretraction of the tissue in the surgical area. Alternatively, theretractor blade stage can allow for the rotational movement only and noradial or flexure movement can occur. In some embodiments, the retractorblade stage can allow for the rotational movement and radial movementonly and no flexure movement can occur. In some embodiments, theretractor blade stage can allow for the rotational movement and flexuremovement only and no radial movement can occur. In some embodiments, theretractor blade stage can allow for the radial movement only and norotational or flexure movement can occur. In some embodiments, theretractor blade stage can allow for the radial movement and flexuremovement only and no rotational movement can occur. In some embodiments,the retractor blade stage can allow for the flexure movement only and norotational or radial movement can occur. In some embodiments, theretractor blade stage can allow for rotational, radial, and flexuremovement of the retractor blades. Moreover, the retractor blade stagecan incorporate any of or any combination of the features and/orembodiments of the rotatable stage or frame discussed herein. Further,in some embodiments, the retractor blade stage can incorporate variousembodiments of retractor blades described herein.

FIG. 7 illustrates embodiments of the distal end of the clip-on flexiblecable that is attached to a retractor blade. As illustrated, the clip-onflexible cable can contain various combinations of cameras, lightingsources, sensors, tracking, and/or other components such as describedherein. In some embodiments, the flexible cable serves as the platformfor these devices (e.g. sensors, lighting, optics, imaging componentsetc.). In other embodiments, the flexible cable or other cable or linesmay connect to a separate platform that supports these devices,including rigid or flexible retractor blades or a separate rigid orflexible platform.

The clip-on flexible cable can be interchangeable to provide fordifferent platforms depending on the desired use or imaging required.The clip-on flexible cable can be fastened to the retractor blades witha fastening member which can include, for example, a clip, a snap, astrap, a screw, a bolt, a nut, a magnet, or any combination of these aswell as any other method that can facilitate convenient attachment. Forexample, attachment can be accomplished in under one minute possiblyless than 20, 10, 5, 3, or 2 seconds per arm and may be accomplished inmore or less than a second or ½ or ¼ second per fastener, which canoccur, prior to or during surgery.

As described herein, the retractor blade can be made of malleable orflexible material or contain joints that allow for movement of theretractor blade. In some embodiments, the clip-on flexible cable can beconfigured to move and bend with the retractor blade allowing formovement of the retractor blade and the attached flexible cable eitherprior to a surgery or procedure or during surgery or a procedure. Rigid,articulated blades with discrete joints, pulled outward via cables foreach DOF to retract tissue, tissue pressure “unretracts” when cables arereleased, as described above with respect to FIGS. 6C-G.

As evident from the discussion above, the imaging assembly can containan array of cameras on, adjacent to, or integrated as part of theretractor components. The flexible cable and retractor blade on a givenretractor arm can incorporate a single camera or a plurality camerassuch as proximal and distal cameras. The cameras can include varioussensor arrays and lenses in a variety of configurations. In someembodiments, the cameras provide three dimensional (stereo) views. Thecameras can, for example include lenses on a sensor array(s) that mimicthe convergence of the eye. The cameras can be tilted to create theconvergence necessary. For example, in some embodiments, a stereo viewcan be formed from two sensors and two lenses. Additionally, the stereocan be a single sensor which can be split with two lenses, which mayhave converging optical axes that mimic the convergence of a surgeon'seyes. The stereo can be used to provide the surgeon or viewer with athree dimensional view of the operating field. For some viewsconvergence is not required, for many times the focal length in terms ofdistance the views can be parallel or nearly so.

FIG. 8 illustrates a front surface of an example retractor blade orflexible cable that can be attached to a retractor blade. In certainembodiments, the retractor blades or flexible cables can have a proximalcamera 8021 which can be located on a proximal portion of the retractorblade or flexible cable, e.g., closer to the entryway into the surgicalsite or the main body of the retractor. The proximal cameras 8021 on aproximal portion of a retractor blade can include monocular, e.g., 16:9,and/or stereo views. The proximal camera(s) 8021 on the proximal portionof the retractor blade can contain larger sensor arrays and moreresolution. In certain embodiments, the retractor blade can have adistal camera 8022 which can be located on a distal portion of theretractor blade, for example, which will be positioned deeper into thesurgical site in comparison to the proximal location. The distalcamera(s) 8022 on a distal portion of the retractor blade can provideoblique views and/or side views and may contain smaller sensor arraysand need to provide less resolution than the proximal cameras which mayprovide a more comprehensive view. The distal camera(s) 8022 on thedistal end of the retractor blade can be small cameras for oblique andside views within the body cavity near the middle or tip of theretractor blades. Nonetheless, the smaller distal cameras may be stereo.Distal cameras with oblique views make seeing hidden areas possible whenline of sight instruments such as operating room microscopes fail.

In some embodiments, the cameras on the retractor blades or flexiblecable can be tilted, for example, upward or downwards or sideways orcombinations thereof. The cameras can be tilted to achieve differentorientations of the camera. For example, in some embodiments, thecameras can be tilted with the use of hydraulic balloon actuated pistonsthereby orienting the camera in different positions relative to theretractor blade. The tilt of the camera can be changed prior to orduring a surgical procedure. For example, the cameras can be positionedon a stage and the stage can be tilted with the use of hydraulic balloonactuated pistons thereby orienting the camera in different positionsrelative to the retractor blade. In some embodiments, the cameras on theretractor blades can be situated on a track that allows the camera tomove vertically (and/or laterally) on the retractor blade therebychanging the position of the camera. Such a vertical position can be setprior to surgery or during surgery. The positioning may be performedmanually or be using an actuator, such as a motor or other actuator.

In some embodiments, the flexible cable or the retractor blades cancontain light sources. The light source can include LEDs and/ormulticolor LEDs. A light source such as a single LED or multiple LEDscan be located on, adjacent to, or integrated as part of the retractorblade or flexible cable. The LEDs or other light source can provideillumination of the surgical area as well as possibly color management.The LEDs can provide sufficient brightness for a surgical treatment. Insome embodiments, multi-color LEDs can provide color balancing of theimage of the surgical area. The intensity of the different colors can beadjusted to provide the desired aggregate color. Through such colormanagement, for example, multi-colored LEDs can be used to control thecolor temperature of the medical lighting. The color managementaccomplished with the different color LEDs can allow for precise controlof the brightness and color of the light emitted from the LEDs. Thecontrol of the color temperature can provide the desired lighting in thesurgical field for the surgeons. In the embodiment shown in FIG. 8 theflexible cable or the retractor blades surface contain an LED 8023 at adistal end of the flexible cable or retractor blade for illumination ofthe surgical area.

As discussed elsewhere herein, the flexible cable or retractor bladescan contain a component which allows for tracking of a location and/ororientation, and facilitate registration of the attached cameras. Theflexible cable or retractor blades can incorporate various methods oftracking including: EM tracker, optical tracking, inertial measurementunits (IMUs), encoders, and other methods as described herein. In someembodiments, the reconfigurable shape of the retractor blade and theflexible cable may increase the usefulness of the tracking for imageprocessing.

As discussed above, platforms other than flex cable can be used.

Flexible Cable

FIG. 3C, described above, however, illustrates an embodiment of aflexible cable 7001 that removably attaches to the retractor blade 7003.In one embodiment, the flexible cable 7001 connects at its proximal endto the aggregator 7007 and has a distal end 7005 that is attached to thefront surface of the retractor blade.

The flexible cable 7001 can contain the camera modules, cameras, LEDs,sensors, and/or other components as described herein as well as anysignal lines or connectors necessary for their varied operations or use.The flexible cable 7001 can contain the electrical signal lines for thecameras, sensors, or other components. Additionally, in someembodiments, the distal end 7005 of the flexible cable can allow formultiple cameras, sensors, trackers, or LEDs to be positioned in variouscombinations and varying numbers. In some embodiments, the cameras canbe distal relative to the surgeon's hand and tool handle. Additionally,in some embodiments, the cameras or camera modules can also be distalrelative to the tool shaft.

The distal end 7005 of the flexible cable can be fastened to theretractor before use through the use of a fastening member which caninclude, for example, a clip, a snap, a strap, a screw, a bolt, a nut, amagnet, or any combination of these as well as any other method that canfacilitate convenient attachment, for example, that can be accomplishedin under one minute possibly less than 20, 10, 5, 3, or 2 seconds perarm and may be accomplished in more or less than a second or ½ or ¼second per fastener, which can occur, for example, prior to surgery. Inone embodiment, the distal end 7005 of the flexible cable can be aplatform which can be separated from the portion of the flexible cablethat connects to the aggregator. The distal end 7005 of the flexiblecable can include all functions and components described herein forclip-on flexible cables and retractor blades such as optics, sensors,LEDs, heaters, trackers, and cameras. The distal end 7005 of theflexible cable can be attached to the portion of the flexible cable thatconnects to the aggregator by a female/male connection port, plug andsocket connection, or other hardware interface that allows an electricalsignal to pass. In some embodiments, the platform 7005 includes a maleconnection for ease of cleaning and sterilization.

The flexible cable can be made of a material which allows the flexiblecable as well as the components on the flexible cable to be rolled orfolded into a compact size. FIGS. 9A and 9B illustrate embodiments of anaggregator 4007, with one or multiple flexible cables 4001 in the rolledconfiguration. FIGS. 10A and 10B illustrate embodiments of theaggregator 4007, with one or multiple flexible cables 4001 in theunrolled configuration. The aggregator 4007 can have one or multipleflexible cables 4001. The flexible cables 4001 can have a single cameraor a plurality of cameras, sensors, LEDs, or camera modules as describedherein.

In one embodiment, the flexible cable can be split up at the distal endcreating multiple extensions. FIG. 11B illustrates an embodiment of theflexible cable 4001 with cutouts forming multiple extensions 4003. Theimaging assembly can have only one flexible cable 4001 with multipleextensions 4004 as shown in FIG. 11C. In various embodiments, themultiple extensions 4003, 4004 can be large enough to fit on a singleretractor blade. Accordingly, in some embodiments, the multipleextensions 4003, 4004 together may have a width smaller than the widthof the blade although the multiple extensions can be wider. In certainembodiments, the flexible cable in the unrolled configuration can havemultiple extensions 4003, 4004 where the extension can have a width suchthat one extension can be placed on a single retractor blade.Accordingly, in some embodiments, one extension may have a width smallerthan the width of the blade. The plurality of extension togetherhowever, may be wider than the blade. In various embodiments similar tothat shown FIG. 11B the multiple extensions 4003 can be small enough sothat all extensions 4003 are placed on the same retractor blade,however, the flexible cable with multiple extensions can have a singlecamera or stereo camera pair on each extension of the flexible cable. Insome embodiments, the flexible cable with multiple extensions can havemultiple cameras on each extension of the flexible cable.

In various embodiments, the flexible cable or other platform can beunrolled by hand. In some embodiments, hydraulics or pneumatics may beused to effectuate unrolling. Fluidic lines in the cable may, forexample, be connected to a hydraulic or pneumatic pump source which canbe applied to cause the cable to unroll and extend.

In various embodiments, the imaging assembly including the aggregatorwith rolled cables can be provided independent of the retractor unit andclipped-on or otherwise attached to the retractor unit prior to use asdescribed herein. The imaging assembly can be attached at a singleattachment point or through multiple attachment points. In variousembodiments, for example, the flexible cable clipped on to the retractorframe, arms, and retractor blades. The flexible cables can be attachedto the retractor and the retractor blades by a separate attachmentmechanism that can be easily attached or removed within seconds. See,for example, FIG. 3C.

FIG. 3B, described above, illustrates an embodiment of the retractorblades with an integrated imaging assembly clipped on to the retractorframe and arms. The retractor blades can be attached to the retractorarms by a separate attachment mechanism that can be easily attached orremoved within seconds. In some embodiments, the aggregator isreleasably attached via an attachment mechanism to the top surface ofthe retractor frame. In some embodiments, the imaging assembly can havemultiple attachment mechanisms positioned on the aggregator, theflexible cable, the retractor blades or combinations thereof whichfasten the imaging assembly to the retractor at various connectionpoints. In certain embodiments, the imaging assembly may have oneattachment mechanism and a single connection point on the retractorframe or retractor blades. As discussed above, the attachment mechanismscan include, for example, a clip, a snap, a strap, a screw, a bolt, anut, a magnet, or any combination of these as well as any other methodthat can facilitate convenient attachment. For example, that can beaccomplished in under one minute possibly less than 20, 10, 5, 3, or 2seconds per fastener and may be accomplished in more or less than asecond or ½ or ¼ second per fastener, which can occur, for example,prior to surgery.

A wide variety of fastening systems may be employed. In variousembodiments, for example, the distal end 7005 of the flexible cable isfastened to the retractor blade surface or the retractor blades arefastened to the retractor frame through a clip-on mechanism 4011 shownin FIG. 12A which can be secured, for example, prior to surgery.

In some embodiments, the clip-on attachment mechanism can also resemblea hairpin attachment 4010 as shown in FIG. 12B. In some embodiments, thehairpin attachment 4010 allows the flexible cable to be releasably andsecurely attached to the front surface of the retractor blade or theretractor blades to be releasably attached to the retractor frame. Extrafastener components may also be included to reduce risk of inadvertentmovement or misalignment. Such clip-on attachment devices may be alsoused for retractors having different configurations such as for example,cylindrical type retractors such as shown in FIG. 21.

Additionally, in some embodiments, the flexible cable can be attached tothe retractor blades or the retractor blades can be attached to theretractor frame via a dovetail attachment. FIG. 12C illustrates anembodiment of a top view of a dovetail attachment mechanism. Theretractor blade or retractor frame may, for example, contain a receivingslot 4012 on the inner surface of the retractor blade or retractor frameopening to the interior of the surgical area. Additionally, the flexiblecable or retractor blade can contain a first and second edge 4013 and4014 configured to fit within the receiving slot 4012. The insertion ofthe flexible cable or retractor blade into the receiving slot 4012 ofthe retractor blade or retractor frame provides a secure attachment ofthe flexible cable to the retractor blades or a secure attachment of theretractor blades to the retractor frame. Further, in some embodiments,the flexible cable can be attached to the retractor blade by insertioninto the receiving slot of the retractor blade and also by a clip-onattachment mechanism as described herein. The locations of the receivingslot and edges that are fit therein can be reversed.

In certain embodiments, the aggregator with rolled cables can bepermanently attached to the retractor frame. In such embodiments, theflexible cable could be rolled out and clipped on to the retractorblades. In some embodiment, the preattached aggregator and flexiblecable can be permanently attached to the retractor unit in the unrolledconfiguration. In some such embodiments, the flexible cable can bepermanently attached to the retractor surface and/or the retractor bladesurface.

A variety of options are thus possible. In certain embodiments, theimaging assembly can be a clip-on imaging assembly which can be clippedon to the retractor and/or the retractor blades. In other embodiments,the retractor blades with integrated components can be permanentlyattached to the retractor frame along with the flexible cable andaggregator. In some embodiments, the retractor blades with integratedcomponents can be permanently attached to the retractor frame but theflexible cable and aggregator can be clipped on to the retractor unit.In such embodiments, the flexible cable would attach to the integratedretractor blades through a hardware interface (e.g., female and maleconnectors, respectively) that allows the electrical signal to pass. Insome embodiments, the retractors can have different shapes. For example,in some embodiments, the retractors have separate arms similar to thatshown in FIG. 3A. In some embodiments, the retractor can be acylindrical retractor such as the retractor shown in FIG. 21.

Surgical Tool with Integrated Camera

In some embodiments, a second surgical device, e.g., surgical tool, maybe used in conjunction with a first surgical device. For example, asshown in FIG. 13A, the first surgical device 100 is a retractor withthree retractor blades, and the second surgical device 200 can be asurgical laser. The laser 200 also includes an integrated camera module205 which provides a field of view determined by the position of thedistal end 211 of the laser. The laser output aperture 213 is orientedin the same direction as the field of view of the camera module 205 ofthe surgical laser 200. In some embodiments, the image obtained from thelaser 200 may be associated with the composite image generated by theplurality of cameras integrated within the first surgical device 100.For example, in some embodiments, the view obtained by camera module onthe laser tool may be superimposed over a portion of the composite imagegenerated from the plurality of cameras, e.g., in the retractor 100. Thesuperimposed image may be positioned so as to indicate the placement ofthe view with relation to the view of the surgical field provided by theplurality of cameras mounted in the first surgical device.

In some embodiments, for example as illustrated in FIG. 13B, the secondsurgical device can be a surgical tool such as a needle holder 300. Invarious embodiments, the surgical tool can be, for example, drill,Kerrison, a cutting tool, grasping tool, Ronguer, scalpel, scissors,forceps, etc. The surgical tool 300 may have an camera module 305integrated therein or attached thereto, whose field of view isdetermined by the position of the tool. For example, CMOS sensors andmicro-optics can be incorporated into both power and non-power tools. Aflex cable 311 (not shown) extends along the length of the tool 300 andelectrically connects to the camera module 305. In various embodiments,the flex cable 311 can provide electrical connection to the cameramodule, as well as supplying gas and/or fluid for camera cleansing, asdescribed in more detail below. The additional camera included with thesurgical tool may have a different magnification than the cameras on thesurgical device, may provide increased resolution, and/or reducedobscuration. As described in more detail below, the video stream fromthe separate camera on the surgical tool can be superimposed over orotherwise displayed with the composite image generated by, e.g.,stitching or tiling images from the plurality of cameras integrated withthe surgical device 100.

The tool image can provide an extreme close-up view of the tool-tissueinteraction, while the wide-field image can provide a stereo image withsituational awareness. The enhanced situational awareness is provided bythe wide field-of-view deep within the area proximal to the surgicalsite, and views of anatomy of interest from more perspectives than asingle device such as a microscope or endoscope. This perspective may befrom within the body if the plurality of sensors on the first surgicaldevice are within the body or at the surgical opening or within a few to75 millimeters therefrom. In some cases, this perspective may beconveniently provide with surgical devices that are attached to the bodysuch as by retractors.

The CMOS sensors, optics, EM sensor (optional), and tool actuation (ifnecessary) may be tightly integrated with the second surgical device inorder to achieve a sufficiently small package. Power tools can be used,particularly for robotically assisted surgery. Embodiments describedherein, however, may be used both in manual and robotic surgery.Electromagnetic tracking sensors, IMUs, and/or one or more cameras maybe incorporated into a wide range of tools, including non-powered tools(e.g., pics, knives, curettes, osteotomes, rasps, trocars, dermatomes,retractors, suction cannulas), manually actuated tools (e.g., scissors,forceps (including bipolar cautery/diathermy forceps), clip appliers,Rongeurs, and needle holders), and powered tools (e.g., drills, powerKerrison, power bipolar forceps, harmonic scalpels, ultrasonic tissueremovers, and lasers). The applicable tools, however, are not limited tothese.

As with the first surgical device, pairs of cameras on the secondsurgical device can provide a stereo or 3D effect.

Power tools can provide an integration of the multiple camera systemwith master-slave surgical robotic systems, as well as decreasing handfatigue and increasing precision with manual surgery. A proportionalfoot pedal control may be used, in which increasing depression of thepedal closes the tool proportionally. In some embodiments, handactuation may be used instead, for example by use of a control lever orpush button. In some embodiments, surgical impedance feedback may beprovided. For example, sensed actuation pressure (or current ifelectrically controlled tool) can correspond to the tissue resistanceand can be used to drive current/torque to foot pedal to providesurgical impedance feedback. Other configurations can be used to providetactile feedback corresponding to mechanical forces experienced by thetools to the surgeons controls such as the foot pedal.

Tool actuation options include electrodynamic, pneumatic, and hydraulic.Hydraulic actuators advantageously are inexpensive, powerful, and“stiff”, meaning that these actuators are less prone to overshoot thantheir pneumatic and electrodynamic counterparts. Hydraulic actuators mayinclude but are not limited to a rolling edge diaphragm (fluid is onlyone side of rolling edge diaphragm), diaphragm with dual actuation(fluid on both sides of diaphragm), as well as a Bourdon tube (axial orhelical), or a piston in a cylinder. Master hydraulic actuator optionsinclude a brushless DC motor with ball screw linear actuator drivingrolling edge diaphragm, and commutated or non-commutated brushless DCmotors driving a rolling edge diaphragm. In various embodiments, duallinear actuators can drive dual air cylinders and air over fluid indisposable cassette. Other configurations are possible.

Tracking Tools and Optics

In various embodiments, cameras can be independently adjustable, eitherrotatable/articulable within the device, or they may be able to beseparately mounted prior to or during insertion of the surgicalplatform. Multiple cameras can each have a position and/or orientationthat is tracked, e.g., electronically or optically. Tracking theposition and orientation of the sensors can provide the image processorwith real-time, low-latency, 6-DOF (six degrees of freedom) informationneeded to decrease search and registration times for the purpose ofstitching a composite image. Tracking can also be employed to assist intiling or otherwise arranging images such that the position of the imageas displayed is consistent with the arrangement of the locations of thecameras or the field-of-views of the cameras in the surgical site.Tracking options include electromagnetic tracking (as, for example,using NDI Aurora or Ascension medSafe). In such systems, 6-DOF sensorsmay be less than 1 mm in outer diameter. These systems do not require aline-of-sight, as in optical tracking systems. Electromagnetic trackingsensors can be easily integrated within the surgical devices atrelatively low cost. In some embodiments, less than 6 degrees offreedom, e.g., 5-DOF tracking can be employed instead of 6-DOF. In someembodiments, for example, IMUs can be used for 5-DOF tracking. In such5-DOF tracking, the distance to target is not required to be precise solong as the other distances and positions are known. If the distance totarget is a few millimeters closer or further away, the difference inmagnification can be negligible. In various embodiments, in which thedistance to target must be precise, 6-DOF tracking may be required.6-DOF tracking may improve integration of the tracking with navigationsystems and the GUI. The capability of tracking, and thereby knowing thelocation and orientation of the camera, can provide the opportunity totell the camera where it needs to be for an array of images to bealigned in a prescribed manner. For example, a plurality of cameras onmalleable retractor fingers by definition are not aligned to an equator.Through tracking, the cameras can offload position derived by sensor andimage processing to position adjustment, described in more detail below.

Surgical tools can also be tracked. Tracking can provide 6-DOF real-timeposition and orientation to image process to enable correct positioningof PIP overlay of tool images or alternatively stitching, tiling, andscaling of images into the composite image. EM tracking has significantadvantages. Since electromagnetic tracking does not rely online-of-sight (as opposed to optical tracking), visual obscuration doesnot interrupt tracking. Particularly in minimally invasive procedures,the presence of various surgical tools within the small incision canprovide significant obstacles to optical tracking. The EM tracking maytake the form of EM sensor coils, or other techniques. In addition thepositions and/or orientations of the cameras and/or the surgical toolscan be tracked by using encoders, MEMS IMUS, ultrasonic emitters,optical tracking or other approaches, in some embodiments.

Electromagnetic tracker coils can be positioned within sufficientproximity to the each camera and the tracking device included therewithsuch that the relative location and/or orientation of the cameras can bedetermined. Such electromagnetic coils can provide 6-DOF position andorientation information, which may then be transmitted to the imageprocessor. The position and orientation information can be used toreduce the computational load required to stitch or tile the variousimages into the composite image and/or to render the image in stereo keyapplication is touch screen user interface. In some embodiments, camerascan be integrated with retractors that include malleable blades, forexample retractors designed for neurosurgery. The position of thecameras individually can therefore provide positional informationunavailable when detecting only the position of the retractor blade,since the blade itself can be malleable. In some embodiments, theretractor and its blades can be rigid, and accordingly the trackingrequirements may be reduced.

In some embodiments, the surgical device and/or any surgical tools caninclude integrated motion sensors, such as gyroscopes or other MEMSaccelerometers. These sensors can measure the physical motion of thecameras due to movement of the device. This motion can be subtractedfrom the image in order to render a displayed image in which the area ofinterest is relatively still, despite any movement by the surgicaldevice.

Camera connectors can have EEPROM chips to provide information to theimage processor about the camera such as the identity, sensor format(e.g., number of horizontal pixels and vertical pixels, etc.) or otherinformation. In the special case of cameras mounted on a tubularretractor for spine surgery or other retractors with fixed cameraposition, the fixed positions may be made available to the system.

If an optical navigation/tracking system is used on retractor,electromagnetic tracking can send tool tracking information to thenavigation system to avoid the need for optical recognition features(for example passive reflections of LEDs and/or location of fiducials)on the tool. Electromagnetic trackers can also be positioned on apatient, for example on a bone or anatomic landmarks, as well as beingpositioned on external fixation systems, for example a cranial fixationsystem for neurosurgery. The position of the cameras with relation tothese external tracked points can be used as input to guide opticalnavigation systems, for example those provided by Medtronic, Stryker, orBrainLabs.

In the case of malleable retractor blades, the position of the camerasintegrated therein are not necessarily fixed with respect to theretractor frame. Accordingly, tracking of the cameras may be useful forenabling an image processor to stitch (register) or tile multiple imagestogether in the proper position and orientation without excessivecomputational load. A 6-DOF electromagnetic sensor can be attached orintegrated within the malleable retractor blade near the camera. In someembodiments, the flex circuits for the camera and sensor coil can beintegrated together or may be potted together and use the same cableassembly.

An alternative approach to tracking camera position can be employed inconfigurations with retractor blades that can flex only in one plane. Anexample of such a 1-DOF retractor blade is shown in FIG. 6A, asdescribed above. With reference to FIG. 6A, the axial position of thewire 353 can be sensed, for example, by using an optical encoder markingon the wire 353. A moving 6-DOF electromagnetic sensor outside thesurgical site also can be attached to the base of the retractor blade351 at the proximal end, e.g., at the position of attachment to aretractor frame. Information from the 6-DOF sensor along with the sensedaxial position of the wire 353 can provide position and orientationinformation about the camera 355. In other embodiments, two wires may beused, one having a distal attachment, the other with a more proximalattachment. For example, the wires may run along the outer curvature ofa pre-flexed Nitinol retractor blade. Axially moving one or both wires,each of which may be sensed, enables incremental controllable flexure ofthe blade. Encoders may be used again for tracking.

In other embodiments, particularly when the retractor blades aresubstantially rigid, the position of the blades can be sensed, and theposition of the cameras can be inferred from the blade position. Forexample, the position of the cameras with respect to the retractorblades on which they are mounted can be known in advance. In use, as theretractor blades are moved, the movement can be tracked. For example,encoders may be positioned within joints of the retractor, such that asthe retractor blade is rotated at the joint, the degree of rotation issensed. Similarly movement of an articulating arm to which the retractorblade is attached can be sensed, and this information can be used toderive the position of the cameras.

It is also possible to use optical tracking, where the surgical deviceincludes some kind of identifying markers, and the information is viewedby an overhead camera. The image can then be processed to identify theposition and orientation of the surgical tool. The markers, whethershape or color or both, and whether positioned on or within a tool, forexample, may provide white balance and intensity information useful foradjusting the sensors or illumination output. In various embodiments,white balancing may be done by inserting the target into the workspacein the field of view of all cameras before surgery. The frame,retractors and assorted cameras could be placed in a holster, whosepurpose is multifold, to include but not limited to: authentication,white balance, camera positions with respect to frame, synching cameratypes to icons in GUI, such as field of view, line of sight, sensortype, stereo pairing, etc.

In various embodiments inertial measurement units may be employed todetermine movement and/or orientation of the cameras. Such inertialmeasurement units may be less expensive than potential alternativetracking options. In certain embodiments, IMUS are used to provide 5-DOFas opposed to 6-DOF.

In some embodiments, the image obtained from the tool may be blurred orotherwise deteriorated due to operation of the tool. For example, asurgical drill with an integrated camera may produce blurred images dueto the rotary motion of the drill. Such image motion can be compensatedby the image processor. For example, the torque produced by the rotationof the drill can be proportional to the applied current. In view of theapplied current, a feed-forward command can be sent to the imageprocessor to compensate for the blur caused by the rotary motion.

In some embodiments, actuation of an electrically powered surgical toolcan result in electromagnetic interference with the electromagnetictracking. In such configurations, optical tracking can be used tosupplement electromagnetic tracking as an alternative. Alternatively, anotch filter can be employed to reduce interference from the poweredsurgical tool with the electromagnetic tracking. The notch filter can beselected such that the stop-band corresponds to the electromagneticnoise produced by the powered surgical tool. The electromagnetic signalused by the trackers may fall outside of the stop-band of the notchfilter. Another approach to avoiding deleterious interference with theelectromagnetic tracking is for electromagnetic tracking to be suspendedduring operation of the powered surgical tool. Once operation of thepowered surgical tool has ceased, electromagnetic tracking may thenre-commence. In various embodiments, a controller can automaticallycease electromagnetic tracking when the powered surgical tool isinitiated, and similarly can automatically resume electromagnetictracking when use of the tool ends. Yet another approach involvescharacterizing the electromagnetic interference caused by a givenpowered surgical tool, and then using that characterization to subtractout the interference from the electromagnetic tracking signal. Forexample, in some embodiments, the electromagnetic signatures for asurgical tool may be known in advance, and this signal may be accountedfor when electromagnetically tracking the positions of the tool and/orcameras. In other embodiments, the electromagnetic noise caused by thesurgical tool may be measured on the fly, and this noise may then besubtracted or otherwise compensated for when calculating the position ofthe surgical tool and/or cameras.

Surgical System Components

FIG. 14 illustrates a block diagram of an example surgical system 370comprising an imaging surgical system 372, a display system 374, and auser interface 376. The surgical system 370 can be used to visualize asurgical site using multiple cameras 378 a and/or 378 b associated witha retractor 380, a surgical tool 382, and/or auxiliary cameras. Visualinformation can be presented to a surgeon using the display system 374to provide visual feedback to the surgeon to enable the surgeon tocontrol the surgical tool 382 using the user interface 376 and/or thetool control 377. The imaging surgical system 372 can be configured toenhance the situational awareness of the surgeon by displaying imagery(e.g., video and/or still images) of the surgical site where enhancementof the surgeon's situational awareness occurs based at least in part onmultiple points of view of the surgical site, an orientation of theimagery on the display, positions of the imagery relative to oneanother, stereo and/or layered imagery, information about a position ofthe surgical tool 382, imagery of the surgical site from externalsources (e.g., MRI, x-ray, CT, or other imaging modality), or anycombination of these. The imaging surgical system 372 can be configuredto provide imagery of a trajectory of the surgical tool 382, forexample, from entry of the surgical tool 382 at a surgical opening todeep within a convoluted surgical site through the use of multiple viewsprovided by multiple cameras 378 a.

The imaging surgical system 372 can include a control system 384configured to receive input from various systems and/or modules, toprocess information, to store data, to send output to various systemsand/or modules, to receive input from a surgeon or other user, or anycombination of these. The control system 384 can include a controller386, data storage 388, a sensor module 390, an image processing module392, a tracking module 394, and a lighting module 396 which maycommunicate with one another and/or external systems throughcommunication bus 385.

The control system 384 includes the controller 386 configured to processdata and to control communication between the control system 384 andexternal systems (e.g., the cameras 378 a, 378 b; the sensors 398 a, 398b; the lights 399 a, 399 b; the display system 374; the user interface376; the tool control 377; a laptop; a tablet; or any other externalsystem). The controller 386 can be configured to control datacommunication between control system modules and/or between controlsystem modules and data storage 388. The controller 386 can beimplemented in hardware, software, firmware, or any combination ofthese. For example, the controller 386 can include logical elementsconfigured to receive imagery from the cameras 378 a, 378 b and performimage processing functions on the imagery according to instructionsprovided by the image processing module 392. As another example, thecontroller 386 can include control modules configured to make decisionsbased on information received from various modules within the controlsystem 384 and/or from external systems. As another example, thecontroller 386 can include one or more physical processors configured toprocess information from the user interface 376 or the tool control 377to send to the surgical tool 382, to the cameras 378 a, 378 b, thelights 399 a, 399 b, or to the display system 374. As used herein, theterm “processor” refers broadly to any suitable device, logical block,module, circuit, or combination of elements for executing instructions.The controller 386 can include any conventional general purpose single-or multi-chip microprocessor such as an Intel® processor, a MIPS®processor, a Power PC® processor, AMID® processor, ARM® processor, or anALPHA® processor. The controller 386 can include any conventionalspecial purpose microprocessor such as a digital signal processor. Thecontroller 386 and the various illustrative logical blocks, modules, andcircuits described in connection with embodiments disclosed herein canbe implemented or performed with a general purpose processor, a digitalsignal processor (DSP), an application specific integrated circuit(ASIC), a field programmable gate array (FPGA), or other programmablelogic device, discrete gate or transistor logic, discrete hardwarecomponents, or any combination thereof designed to perform the functionsdescribed herein. The controller 386 can be implemented as a combinationof computing devices, e.g., a combination of a DSP and a microprocessor,a plurality of microprocessors, one or more FPGAs, one or moremicroprocessors in conjunction with a DSP core, or any other suchconfiguration. In some embodiments, the controller 386, the sensormodule 390, the image processing module 392, the tracking module 394,and the lighting module 396 or any combination thereof can beimplemented using one or more FPGAs. In some embodiments, the controller386 and one or more modules can be implemented using one or more FPGAswith associated components. For example, in certain implementations, anFPGA can be configured to provide image processing capabilities as wellas video switching, controlling video sent between components of thesystem 370, controlling data processing, and/or controlling a flow ofdata between components of the surgical visualization system 370.

The control system 384 includes data storage 388. Data storage 388 canbe coupled to the other components of the control system 384, such asthe controller 386, the sensor module 390, the image processing module392, the tracking module 394, and the lighting module 396. Data storage388 can refer to electronic circuitry that allows information, typicallycomputer data, to be stored and retrieved. Data storage 388 can refer toexternal devices or systems, for example, disk drives or solid statedrives. Data storage 388 can also refer to fast semiconductor storage(chips), for example, Random Access Memory (RAM) or various forms ofRead Only Memory (ROM) such as an EEPROM, which are directly connectedto the one or more processors of the control system 384. Other types ofmemory include bubble memory and core memory. In some embodiments, videooutput from the image processing module and/or other data fromcomponents of the surgical visualization system 370 can be stored ondata storage 388. For example, video displayed on the display system 374can be recorded onto data storage 388. In some embodiments, data storage388 is a removable data storage system, such as an external USB harddrive, a flash drive, a memory card, or other similar data storagedevice.

The control system 384 includes sensor module 390 configured to receiveand process information from the sensors 398 a, 398 b associatedrespectively with the retractor 380 and the surgical tool 382. Sensorscan include inertial measurement units (IMUs), gyroscopes,magnetometers, accelerometers, thermal sensors such as thermocouples orthermistors, electromagnetic sensors, photosensors, or other suchsensors. In some embodiments, the sensor module 390 receives sensorinformation and processes it to provide feedback to the surgeon throughthe user interface 376, the tool control 377, the display system 374, orthrough some other method such as audible or visual signals. Forexample, the sensor module 390 can receive accelerometer informationfrom a sensor associated with the surgical tool 382 and the imageprocessing module 392 can use this information to display a position ofthe surgical tool 382 when outputting imagery to be displayed on thedisplay system 374. In some embodiments, the sensor module 390 receivessensor information and processes it to provide sensor information, forexample, to the image processing module 392, the tracking module 394,and/or the lighting module 396. In this way, the control system 384 canuse the sensor information to alter operational parameters or control ofthe surgical system 370 without direct intervention from a user orsurgeon. For example, the sensor module 390 can receive temperatureinformation from a thermocouple associated with the surgical tool 382 orcamera 378 a and the control system 384 can decide to implement acooling procedure if the temperature exceeds a threshold. In someembodiments, sensor information received by the sensor module 390 isstored in data storage 388 for later use and/or analysis. Other uses ofsensors in conjunction with operation of the surgical system 370 aredescribed herein.

In some embodiments, the retractor 380 and/or surgical tool 382 can beconfigured to provide an encrypted signal to the control system 384, forexample, to the sensor module 390 or the image processing module 392. Insome embodiments, similar to the encrypted signal, the retractor 380and/or surgical tool 382 can be configured to provide a header on imagedata packets to the control system 384, for example, to the sensormodule 390 or the image processing module 392. Encryption or the headercan be provided by the camera chip or a separate chip electricallyconnected to the optical sensor in the camera. In some embodiments, theheader or the encryption of the signal assists the signal processingsystem in the recognition of the camera once attached or clipped on tothe retractor. The header or encryption may also ensure the properpairing of imaging assembly and retractor base for the appropriate imageprocessing and imaging system configuration. Additionally, the header orencryption can ensure the proper standard and quality of camera is usedand complies with appropriate requirements for the image processingherein described. In various embodiments, decryption can be achievedwith, for example, a computer processor or other electronics, possiblythe image processing module or other signal processing or computingmodule. In various embodiments, the header can be used to relaynon-image information to the various systems for use in control thesystem, processing imagery received from cameras, displaying outputvideo or images, and the like.

The control system 384 includes the image processing module 392configured to receive image data, process the image data, and to outputvideo or image data for display. The image processing module 392 can beconfigured to receive image data from the cameras 378 a and/or 378 band/or from other imaging modalities including MRI, CT, X-ray, and thelike. In some embodiments, image data from other imaging modalities canbe communicated using, for example, the DICOM3 protocol. The cameras 378a, 378 b can include optics, optical sensors such as CCD or CMOStwo-dimensional detector arrays, and associated electronics configuredto acquire image information of a scene, as described herein withgreater detail. The image processing module 392 can be configured toprocess image data which can include, for example, stitching images,blending images, morphing images, tiling images, performingtransformations on the images (e.g., affine transformations, non-lineartransformations, etc.), mosaicing images, enhancing resolution of aregion using a plurality of images, extracting surgical tool 382position information, forming stereo images from separate monocularimages, or any combination of these or other image processing includingbut not limited to those described herein.

The image processing module 392 can be configured to output video orimages for display on the display system 374, the user interface 376, orboth. In some embodiments, the output video or images can include videofrom a single camera or a composite video where the video is a result ofcombining imagery from a plurality of cameras. For example, imagery frommultiple cameras can be combined to form a single video by stitching thevideos to create a video stream having a wider field of view thanprovided by any of the multiple cameras used in the stitching. Asanother example, video from multiple cameras can be combined to enhancea resolution of a region of interest such that the resulting videopresents a greater number of pixels per unit of area than any individualinput video frame. As another example, multiple cameras can providemultiple viewpoints of an object or area and the video from the multipleviewpoints can be morphed to create video from a virtual viewpointdifferent from any of the viewpoints of the multiple cameras.

In some embodiments, the image processing module 392 outputs video orimage data for display where the output includes a plurality of imagesor video. For example, the image processing module 392 can receiveimagery from a plurality of cameras and present these as tiled images orvideo. In some implementations, the output tiled images or video can beconfigured to represent information about the input imagery. Forexample, one or more of the tiled images or video can be presented as atrapezoid or other similar shape to represent an orientation of thecamera relative to an object or area being imaged, e.g., to representthat the focal plane of the camera is not parallel to the area or objectbeing imaged. As another example, the output tiled images or video canbe positioned to represent their physical arrangement which can include,for example, the relative positions of the cameras providing the imageryand/or the field-of-views of the cameras. In certain implementations,the tiled images or video can be presented with borders around one ormore of the images to enhance or help to identify an extent of thecorresponding video or images.

In some embodiments, the image processing module 392 outputs acombination of any of individual, tiled, and/or stitched imagery. Forexample, in certain implementations, the image processing module 392outputs background imagery having a relatively wide field of view wherethe background imagery is provided by a single camera or where thebackground imagery is a result of stitching imagery from a plurality ofcameras. In some implementations, overlaid on the background imagery,the image processing module 392 can output video having a narrower fieldof view than the background imagery. For example, the video with anarrower field of view can be a combination of monocular imagery thatform a stereo video, an example of which is illustrated in FIG. 15. Asanother example, the imagery with a narrower field of view can beimagery provided by a camera 378 b on the surgical tool 382, an exampleof which is illustrated in FIG. 16. In some implementations, overlaid onthe background imagery is imagery from a plurality of cameras where theimagery can be tiled and/or stitched, an example of which is illustratedin FIG. 16. In some implementations, the image processing module 392 canoutput a center image or video that can change between imagery providedby camera(s) 378 b on the surgical tool 382 and imagery provided bydistal camera(s) 378(a) on the retractor 380. For example, imagery fromthe surgical tool camera(s) 378 a can be presented as apicture-in-picture overlaid on background imagery and/or with stitchedor tiled imagery from retractor camera(s) 378 a, an example of which isillustrated in FIG. 17. In some embodiments, imagery from threeretractor cameras 378 a can be displayed as tiled imagery overlaid onbackground imagery and imagery from the surgical tool camera 378 b canbe displayed as a picture-in-picture-type display over the backgroundimagery.

In some embodiments, the image processing module 392 is configured toreceive imagery from the cameras 378 a, 378 b and to display thereceived imagery for simultaneous viewing on the display system 374. Insome embodiments, the image processing module 392 is configured to allowa system or user (e.g., a surgeon, an assistant, etc.) to manipulate orinteract with the imagery. For example, the image processing module 392can be configured to provide electronic zoom or magnification accordingto user input, to system configuration, to camera position, to surgicaltool movement or position, or the like. In some embodiments, the imageprocessing module 392 provides feedback to the cameras 378 a, 378 b tochange a focus, a viewing angle, a zoom factor, or the like. Feedbackcan be provided by the control system 384, external systems, and/or thesurgeon or other user through the user interface 376. For example, thesurgeon can use the user interface 376 to select images and/or video toview on the display system 374. The control system 384 can receive inputfrom the surgeon, receive imagery from cameras 378 a and/or 378 b,process the received imagery, and send the processed imagery to thedisplay system 374. Functionality and capabilities of the imageprocessing module 392 and image processing of the imaging surgicalsystem 372 in general is described herein.

The control system 384 includes tracking module 394 configured toprovide and track a location of the retractor 380 and/or the surgicaltool 382 during use of the surgical system 370. The tracking module 394can be configured to receive information from the sensors 398 a, 398 b,the sensor module 390, the image processing module 392, the userinterface 376, and/or the tool control 376. The tracking module 394 canprocess this information to calculate a position of the surgical tool382. The tracking module 394 can provide this information to the imageprocessing module 392 for display on the display system 374. Forexample, the image processing module 392 can receive surgical tool 382position information from the tracking module 394 and display thesurgical tool 382 on the display system 374 such that the surgical tool382 is rendered along with imagery of the surgical site in such a way asto provide situational awareness to the surgeon. Surgical tool 382rendering is described herein with greater detail. The tracking module394 can incorporate electromagnetic tracking information, EEPROM,radio-frequency identification (“RFID”) information, accelerometer data,gyroscope data, and other such data to calculate a position of thesurgical tool 382 and/or other components of the surgical system 370. Insome embodiments, the tracking module 394 provides coarse positioninformation of the surgical tool 382 to the image processing module 392which can be configured to use the coarse position information tocalculate fine position information based at least in part on imagery ofthe surgical tool 382.

The control system 384 includes lighting module 396 configured tocontrol illumination of the surgical site. The lighting module 396 cancontrol the lights and/or other sources of illumination 399 a, 399 b ofthe imaging surgical system 372 to provide sufficient illumination ofthe surgical site for the cameras 378 a, 378 b to acquire imagery of thesite. In some embodiments, the lighting module 396 times or coordinatesoutput of the lights 399 a, 399 b such that the lights do not directlyilluminate one or more cameras 378 a, 378 b, as described herein ingreater detail. The lights 399 a, 399 b can be LEDs or other suchsources of visible, infrared, and/or ultraviolet light, as describedherein with greater detail.

The imaging surgical system 372 includes retractor 380 and/or surgicaltool 382, both of which are described in greater detail herein. Theretractor 380 and/or surgical tool 382 can have sensors 398 a, 398 b,cameras 378 a, 378 b, lights 399 a, 399 b, and/or other elements (e.g.,a heater) associated therewith each of which can be controlled using thecontrol system 384, the user interface 376, the tool control 377, orother control mechanism.

The surgical system 370 can include a display system 374 configured todisplay information to a user or surgeon. The display system 374 isdescribed herein with greater detail. The surgical system 370 caninclude a user interface 376 configured to receive input and providefeedback to a user. The user interface 376 can include physical elementsconfigured to receive user input such as, for example, a multi-touchscreen, buttons, keyboards, pointer device, switches, knobs, and thelike. The user interface 376 can include a display configured to providevisual information to the user and to facilitate interaction with thesurgical system 370 and control of the system 370. In some embodiments,the user interface 376 can be incorporated into the display system 374such that it both displays information and receives user input. In someembodiments, the user interface 376 displays the graphical userinterface on a separate display that is not a part of the display system374. In some embodiments, the user interface 376 includes a touch screeninterface and/or a gesture recognition system. The user interface 376can be used to, for example, select imagery to view, zoom-in on imagery,and/or position imagery for display on the display system 374. In someembodiments, the surgical system 370 can receive input in the form ofvoice commands.

In some embodiments, the display system 374 and the user interface 376can be combined in a way that allows the surgeon or other user tointeract with displayed imagery. In some embodiments, the display system374 is configured to provide an immersive experience for a user. Forexample, a virtual display can be used wherein the surgeon viewingdisplays through binoculars motions with his or her hand in a mannerconsistent with the imagery seen by the surgeon, and the surgeon'smovements are detected. These movements can be mapped to actions thatcorrespond to manipulating output imagery, cameras, surgical tools, orany combination of these. In some implementations, the binocularpresentation can allow for three-dimensional information to be displayedto the user and the user can use hand gestures at the point in spacewhere imagery appears to be to initiate a command. In some embodiments,the gesture recognition can be used in conjunction with a surgical tool382. For example, the surgeon can remove the surgical tool 382 from asurgical field and move the tool 382 in such a way that the gesturerecognition system correlates the movement to pointing, flicking,dragging, pointer control, and/or stylus functionality. The displaysystem 374 can include image-based or electromagnetic-based motionsensors to detect the gestures of the surgeon. The tracking system usedto track the tool, for example, may be used. Such a display system 374and user interface 376 can allow for a surgeon to interact with the userinterface 376 without potential contamination issues that may arise whenusing a touch-based user interface.

The user interface 376 can include a graphical user interface configuredto facilitate interaction with the surgical system 370. The graphicaluser interface is described with greater detail below with reference toFIGS. 24 and 24B. In some embodiments, the display system 374 displaysthe graphical user interface 376. The user interface 376 can include thetool control 377 where the tool control 377 can be integrated into aunitary user interface 376 or the tool control can be a separate elementof the user interface 376. The tool control 377 can be configured toreceive input from a user to manipulate the surgical tool 382. Examplesof the tool control 377 and associated surgical tools include thosedescribed herein with reference to FIGS. 34-53.

Video and Image Control and Processing

Surgical visualization systems described herein can include componentsconfigured to receive image data, process the image data and other data,and output video and/or images for display. The video and imageprocessing functionality can be provided using any suitable combinationof hardware, firmware, and software, such as but not limited to thosedescribed herein with reference to FIG. 14. For example, a surgicalvisualization system can include communication buses and/or interfacesto receive image data and one or more micro-processors, FPGAs, ASICs, orthe like or combinations thereof that are configured to process receivedimage data to provide output video for display. For ease of description,video and image processing will be described as being performed by animage processing system which can be any suitable system or component ofa surgical visualization system that is configured to receive imagedata, process image data, and output video or images for display, suchas for example the image processing module described herein withreference to FIG. 14. As part of the description, the image processingsystem is described as receiving and/or outputting imagery, images,and/or video. The processing of images described herein should beunderstood to be applicable to video. Any description of processinglimited to either images or video should not be construed as limitingthe disclosure of the functionality solely to images such as stillimages or video as video can be interpreted as a series of images.Likewise often the term video images or video image, feed or stream orsimply images are used in connection with discussion of video.

In some embodiments, the surgical visualization system includes an imageprocessing system configured to output a single image or video streamwherein the single output image is created from a plurality of inputimages. For example, the image processing system can be configured toreceive a plurality of images from cameras associated with a retractor,a surgical tool, another surgical device, or any combination of these.The image processing module can be configured to combine the pluralityof images into a single displayed scene. For example, the imageprocessing module can be configured to stitch the images to create asingle larger displayed scene by combining images from cameras where theimages overlap. In some embodiments, one or more images may besuperimposed on or shown forward the stitched image etc. For example,this configuration includes possibly picture-in-picture (PIP) as well asan arrangement of tiled images disposed on the stitched images.

Accordingly, the image processing module can be configured to stitchmultiple images together to create a single image. Stitching images canprovide a wider field of view or field. Stitching images can reduce oreliminate obscuration of a region of interest in the scene by a surgicaltool or other device. Stitching images can reduce or eliminate albedodue at least in part to the multiple cameras in the surgicalvisualization system which provides the surgeon the ability to “lookbehind” protruding features, such as an aneurysm. Stitching images canreduce image artifacts such as vignetting caused by a retractor tube orblade. These same benefits may be obtained by other arrangements of aplurality of images from different cameras such as tiling images fromdifferent cameras.

The image processing module can be configured to adjust input imageswhen stitching them together to provide a substantially seamless outputimage. In some embodiments, direct (e.g., pixel-by-pixel) orfeature-based image processing can be used to align images where theyoverlap. Feature-based image processing can include aligning imagesbased on features within the images. This can include image or shaperecognition and matching to align the images. In certainimplementations, a target structure with alignment features can beincluded in a field of view of one or more cameras to assist infeature-based alignment methods. Such a target structure may be providedduring initial set-up. The target structure can also includewhite-balancing features, as described in more detail herein. The targetstructure may be included on the retractor, might be a component addedto the retractor during initialization, or may be a feature of a base,pod, station, or other platform in which or on which the retractor isattached, e.g., rests, during an initialization phase. Similarly,pixel-by-pixel alignment processing can be used, for example, inconjunction with position and orientation information to align pixelsbased on pixel features to stitch the images together. In someembodiments, stitching images can be simplified where the cameras areidentical or have identical sensors, where the relative positions and/ororientations of the cameras are known with relative precision (such byusing memory devices or tracking systems as describe above), and theirdistance to the imaged scene is substantially the same. In someimplementations, the cameras can be positioned near a non-planar scene,or a scene that is not flat, and non-linear transformations can be usedto stitch images. In some implementations, it may be preferable to usecentralized cameras, or cameras that provide imagery of a centralportion of a scene (where central can be relative to the output image),as reference images in stitching to reduce cumulative errors across anoutput image when compared to using a peripheral image as a reference.

When stitching images, several techniques can be used to blendoverlapping regions of images to provide a substantially visuallyseamless output image. For example, the image processing system canimplement an algorithm to blend image intensities at the seams ofstitched images. The algorithm can include blending images usingcenter-weighted averaging (or feathering) wherein pixels near the centerof the image are weighted more heavily than pixels near the edge. Thealgorithm can include using Laplacian pyramid blending and gradientblending to address differences in gain between images at the seamswhich can result in less blurring than center-weighted averaging. Thealgorithm can include averaging in the radiance domain appropriatetypically where gain differences between images are relatively large. Insome embodiments, the image processing system is configured to performaffine transformations on received imagery.

The image processing system can be configured to morph multiple imagesfrom cameras having different viewpoints to create a single image of ascene wherein the single image appears to include a viewpoint that maydiffer from the viewpoints of the cameras. For example, the imageprocessing system can receive a first image of a region of interest froma first viewpoint and a second image from a second viewpoint of a regionof interest and generate intermediate images from viewpoints that liebetween the first and second viewpoints. For example, as illustrated inFIG. 18, left and right cameras can be combined to provide a centralview from a virtual camera. This approach can be performed for any pairof cameras that image a common region. In some embodiments, more thantwo cameras are used to reduce or eliminate peripheral ambiguitiesrelated to imaging 3-D scenes. This technique can provide, for example,a central view of the surgical site without a central camera,microscope, or endoscope. This technique can be used to simulate cameramovement, to provide image data from a viewpoint that is not physicallyaccessible but desirable, to increase image information of a region ofinterest, or any combination of these. Image morphing can, for example,be accomplished using standard techniques that are known in the art.

The image processing system can be configured to display multiple imagesas individual images where the multiple images are tiled. The tiledimages can be purposely tilted and/or distorted based at least in parton an orientation of the camera providing the images. The tiled imagescan be presented with borders around each image or groups of images.Tiling images can be differentiated from stitching or morphed imagesbecause tiled images are not combined to form a unified image of an areaof interest or object of interest. However, in some implementations,images can be stitched or morphed and these stitched or morphed imagescan be output as tiled images. For example, these morphed or stitchedimages can be included together with other images in a tiled fashion.

In some embodiments, positions of cameras can be provided to the imageprocessing system to be used when stitching, morphing, or tiling imagesfrom the cameras. In certain implementations, the cameras can haverelatively fixed positions, such as when the cameras are associated witha tubular spine surgery retractor. The positions of the cameras can becommunicated to the image processing system using any tracking systemsand/or memory storage devices (e.g., where the positions of the camerasare fixed). For example, EM or optical tracking, one or more inertialmeasurement units (IMUs), EEPROMs and/or RFID tags, associated with theretractor, surgical tool, and/or with the cameras may be used. Incertain implementations, for example, the cameras can have EM trackers,optical trackers, IMUs etc which communicate their positions and/ororientation during setup to the image processing system. Suchinformation is especially useful if the camera is movable and the camerais reoriented during set up. In some implementations, RFID or EEPROMtechnology can be used to communicate information about camerafield-of-view, resolution, quantity of cameras in use and possiblylocation and/or orientation. The image processing system can use thistracking information to assist in aligning the images from the variouscameras to improve and/or facilitate stitching, morphing, and/or tilingimages. In some implementations, the tracking information can provide acoarse position of the cameras and their relative pointing angles, andthe image processing system can use this information to calculate atransformation for each received image to stitch them together toprovide a substantially seamless display of stitched images. The imageprocessing system, also, can provide greater detail about the positionsand/or orientations of the various cameras to the surgical visualizationsystem. Even if the images are not stitched, the images can be arrangedin a manner that is consistent with the arrangement and location of thecameras in the surgical field, e.g., on the retractor or theirrespective field-of-views.

The image processing system can be configured to combine images from aplurality of cameras to virtually enhance capabilities of the cameras inthe surgical visualization system. For example, the image processingsystem can create an image of an area of interest having enhancedresolution by combining multiple images of the area of interest. Asanother example, the image processing system can create a virtual camerahaving an increased dynamic range by combining image data from camerashaving differing gains.

In some embodiments, the image processing system can be configured tocombine multiple substantially overlaid images to output a single imagehaving a higher resolution than any of the multiple images. In certainimplementations, enhancing the resolution in this manner results in anoutput image where the number of pixels per unit area imaged is greaterfor the output image than any of the multiple images. In certainimplementations, the multiple substantially overlaid images are providedby multiple cameras having fields of view that substantially overlap. Insome embodiments, images from cameras having different gain are alignedto provide an aggregate image having increased dynamic range.

The image processing system can be configured to provide electronic zoomand/or magnification using image data from a plurality of overlappingcameras. In some embodiments, cameras can be positioned at differingdistances from an area or object of interest. As a result of thesediffering positions, differing views and/or magnifications of the areaor object of interest are provided by the cameras. In some embodiments,electronic zoom may be used to normalize magnification levels. Forexample, images from cameras at different positions can have differentmagnifications and the image processing system can electronically adjustmagnification levels when, e.g., stitching, tiling, and/or morphing theimages prior to outputting the images. In some embodiments, the imageprocessing system super-positions images to permit electronicmagnification of objects to yield better quality and/or similar-sizedimages of an area of interest for display. In some embodiments, imagesthat are overlapping can be electronically magnified such that theresulting magnified images provide adjacent views rather thanoverlapping views.

The image processing system can also utilize the differingmagnifications resulting from cameras located at different distances toelectronically zoom imagery of an area or object of interest. Thesevaried levels of magnification can also be used by the image processingsystem when morphing images. Cameras that are closer to the area orobject of interest present imagery having increased resolution of thearea or object of interest which can be advantageous for electronicallyzooming. The image processing system can use this information to provideelectronic zoom capabilities and magnification capabilities. In someembodiments, the image processing system uses superposition of imageshaving differing levels of magnification to provide electronic zoomcapabilities based at least in part on the different magnificationlevels. Accordingly, in some embodiments, the image processing systemprovides a zoom feature for magnification over an array of camerashaving unequal or equalized image magnifications. In some embodiments,the image processing system rotates images to enable this variablemagnification and zooming capability.

In some embodiments, the image processing system can use camera locationinformation to create a distance guide (e.g., a look-up table) tomagnify or de-magnify images corresponding to an area of interest. Insome embodiments, the image processing system can use the locationinformation to calculate a magnification of a camera and use thisinformation when combining images from multiple cameras having differingmagnifications. In some embodiments, the image processing system can beconfigured to present images from cameras with different levels ofmagnification to represent differing distances from an area or object ofinterest, for example when tiling or with picture-in-picture.

In some embodiments, cameras can include MEMS technology to provide zoomfunctionality by moving and/or positioning the camera optics. The zoomfunctionality can be used to alter magnification of the images acquiredby a camera. The surgical visualization system can include modules andsystems that provide feedback to the cameras such that the camerachanges magnification levels by changing a focal length or zoom of thecamera. For example, the tracking module can provide information aboutthe position of the surgical tool to the image processing system and theimage processing system can provide feedback to the camera to alter itsmagnification factor according to the surgical tool's distance from atargeted tissue site. Similarly, in some embodiments, cameras caninclude MEMS technology to provide focusing functionality by movingand/or positioning the camera optics. The focus functionality can beused to change a focal length of the camera optics to account fordistance to a targeted site. For example, if the depth of focus of acamera on a surgical tool is not sufficient to maintain focus oftargeted tissue when the position of the surgical tool changes, the MEMSsystem can change the focus of the camera. Thus, as described herein,the surgical visualization system can include methods of electronicallyzooming and methods of physically zooming cameras.

In some embodiments, the surgical visualization system can includecameras at different locations within a surgical site, and particularlyat different depths within the surgical site. The image processingsystem can use the varying depths of field from the cameras at differentdepths to increase an output depth of field to provide focused imageryof a greater portion of the surgical site. For example, a retractor canhave two or more cameras or camera pairs at two or more longitudinaldistances along the length of the retractor (e.g. proximal and distal)and the cameras at each level can provide image data to the imageprocessing system. For example, a retractor can have two or more ringsor arrays of cameras (e.g. proximal and distal) located on the retractorso as to provide different distances to the same object in the surgicalsite. The proximal cameras may provide a main larger field of view ofthe area in the surgical site and the distal cameras may provide acloser-up view(s). The image processing system can combine overlappingimage data to provide an output image having a greater depth of focus.An example of a system with cameras located at two depths orlongitudinal positions along the length of the retractor is illustratedin FIG. 21. Locating cameras at varying distances in the surgical sitemay potentially provide for increased situational awareness. Locatingcameras at varying distances and positions can provide for continuousviewing of a surgical site, and, in some embodiments, the continuousviewing can be provided without a need for repositioning cameras duringsurgery. For example, where tools or work obscures one or more cameras,imagery from non-obscured cameras can be utilized to provide a view ofthe surgical site.

In some embodiments, image processing can be used to track a positionand/or orientation of a surgical tool. For tracking, the surgical toolcan include distinguishable or identifiable coloration, patterns,markings, etc. such that the image processing system can identify thesurgical tool, calculate its position within the surgical site, and/orcalculate an orientation of the surgical tool.

In some embodiments, the image processing system receives imagery datafrom one or more cameras positioned on a retractor and processes thisinformation to identify the surgical tool and prepares a video or imagewith a model of the surgical tool positioned and/or oriented within theimage to represent the position and orientation of the surgical tool.For example, optical tracking of the surgical tool can be accomplishedusing cameras positioned on a frame of a retractor wherein the surgicaltool includes, for example a pattern, e.g., a greycode possibly appliedusing laser marking with a known pattern around the tool axis. The imageprocessing system can process the images from the cameras to identifythe greycode on the surgical tool. Using this information and possiblyadditional information from one or more other tracking systems or storeddata, the image processing system can identify a location and/ororientation of the surgical tool within the surgical site.

Accordingly, in some embodiments, a displayed image can be related to aposition of the surgical tool. The image processing system can receivetracking information and/or it can extract position information fromimagery and configure the output imagery (e.g., position or orient theimage relative to a background image or other imagery being displayed)that corresponds to where the surgical tool is currently positioned. Theimage processing system can be configured to track movement of thesurgical tool and integrate this information to smoothly track aposition of the surgical tool on the display (e.g., by changing outputimagery such that imagery of the surgical tool is presented, forexample, as centered on the display system or has a changing locationwith respect to the background or wide field of view main view remainsgenerally still). The image processing system can also change whatimages are displayed, for example, tiled, depending on the location ofthe surgical tool. The image processing system can establish, forexample, a centroid of movement for the surgical tool and based on thiscentroid it can determine whether a movement can be ignored or followed.Such a decision can be based at least in part on dimensions of thesurgical site and/or the retractor in use. In addition, the imageprocessing system can be configured to ignore movements that areoutliers compared to the centroid distribution, such as when thesurgical tool is completely removed from the surgical site.

In some embodiments, the image processor can output imagery of thesurgical site wherein the output imagery includes a rendering of thesurgical tool positioned and oriented within the surgical site accordingto a measured position and/or orientation. In some embodiments, therendering of the surgical tool can have a level of opacity that can varyfrom being completely transparent to being completely opaque and includevarying levels of partial transparency/partial opacity therebetween, asdescribed in more detail herein. In some embodiments, the surgical toolcan be rendered, e.g., in a picture-in-picture, in a display, asdescribed herein wherein the surgical tool can have a configurable levelof transparency (e.g., the surgical tool can be presented as opaque orsemi-transparent). By presenting the surgical tool with a level oftransparency, a user can view underlying imagery of a surgical site. Thetransparent rendering described herein can help to overcome a problem insurgical visualization systems where a surgeon's view of the surgicalsite is obscured by the surgical tool. This functionality can beprovided where multiple cameras acquire images of the surgical site frommultiple viewpoints or using a single camera with the position of thesurgical tool provided by a tracking system. To provide a rendering ofthe surgical tool, a computer model (e.g., a CAD model) of a surgicaltool can be loaded into the surgical system during initialization or atsome other point prior to surgery. In some embodiments, the user canselect whether to display the rendering of the surgical tool. In someembodiments, an icon, graphic, or indicator can be provided on agraphical user interface that allows the user to control the renderingof the surgical tool, such as to choose a level of transparency andwhether to display the rendering.

FIGS. 15 to 17 illustrate example display outputs of the imageprocessing module, suitable for sending to a display system. In someembodiments, the image processing module can be configured to receivevideo from the plurality of cameras at different positions, process thereceived video, and output the processed video to the display. Theoutput video can be stitched, tiled, superpositioned, or otherwisecombined on the display. The arrangement and/or configuration of theoutput video can be automatically produced, manually configured, orboth. In this way, the surgical visualization system can be configuredto display video from multiple cameras for simultaneous viewing. In someembodiments, the display system can be configured to receive processedvideo from the image processing system and position the output videoappropriately with respect to each other for display.

As illustrated in FIG. 15, the surgical visualization system can includeone or more proximal cameras configured to provide a relatively widefield of view of a surgical site. This can provide a background or mainview on a display showing the surgical site. The wide field perspectivecan provide to a surgeon a frame of reference for various areas orobjects of interest to enhance a surgeon's situational awareness. Incertain embodiments one camera or camera pair can be to provide thebackground or main view. In some embodiments, however, the imageprocessing system can provide a wide field of view, for example, bystitching and/or morphing monocular wide field of view camera data, forexample, from multiple, e.g., proximal cameras. In some embodiments, thesurgical visualization system can include one or more, e.g., proximal(or distal), cameras configured to provide stereo imagery of at least aportion of the surgical site, wherein the cameras acquiring stereoimagery can have a field of view that is less than the field of view ofthe wide field of view cameras. The image processing system can providestereo data from adjacent proximal (or distal) cameras or atwo-dimensional sensor array with left and right portions dedicated forleft and right eye (e.g., each with its own imaging optics) and cansuper-position the stereo data on the wide field of view or backgroundview. Accordingly, in some embodiments, the display can include acentral stereo image overlaid on a wide field of view background imageproviding peripheral vision information. In some embodiment thewide-field of view background or main view can be stereo and formed fromimages from proximal (or distal) cameras. Images from the stereo camerascan be displayed on separate displays (e.g. one for each eye) or on thesame display (e.g., which is visible to the left and right eye atdifferent times) to create a 3-D visual effect, as described in moredetail herein.

As illustrated in FIG. 16, the surgical visualization system can includeone or more distal cameras providing other, possibly smaller, fields ofview of the surgical site, where the distal cameras are positionedfurther within the surgical site. In some embodiments, the distalcameras are configured to provide oblique or side video of the surgicalsite and can be displayed as tilted to help the surgeon visuallyinterpret the video they provide, such as by providing depth clues tothe surgeon. In some embodiments, the video received from the distalcameras can be stitched, tiled, tilted, or otherwise processed anddisplayed overlaying or overlaid with the video from the proximal widefield of view imagery and/or the proximal stereo imagery.

In some embodiments, a plurality of the distal (or proximal) cameras, ifnot all, are directed at substantially similar inclinations with respectto the surgical device (e.g., retractor) with which they are associated.In some embodiments, the distal (or proximal) cameras are directednormal to an axis of the surgical device. For example, the retractor mayhave the shape of a right circular cylindrical and have an axis throughthe center of the circular cross-section. The cameras may be directedtoward each other and normal to this axis. Alternatively, the camerasmay be directed downward into the surgical site not normal to this axis.The plurality of distal cameras, however, may have substantially similarinclinations or declinations with respect to the axis. In someembodiments, the inclination or declination with respect to the axis canbe less than or equal to about 30 degrees, less than or equal to about45 degrees, less than or equal to about 70 degrees, and/or less than orequal to about 110 degrees. In some embodiments, the inclination ordeclination with respect to the axis can be greater than or equal toabout 0 degrees, greater than or equal to about 30 degrees, greater thanor equal to about 45 degrees, and/or greater than or equal to about 70degrees.

In some embodiments, a display of information from distal cameras can beactivated according to criteria. For example, initially the proximalcamera(s) may be activated and images produced by these proximal camerasdisplayed. When a tracked surgical tool is inserted into the patient andis at or near a proximal camera at a beginning of a procedure, monocularside views provided by the distal cameras can be displayed on eitherside of the central stereo view in the display, as illustrated in FIG.16, to provide the surgeon with views of where the tool is to be placed.In some embodiments, the views provided by the distal cameras can bedisplayed as tiled images. Additionally as the surgeon or user proceedsdeeper within the surgical site, the distal camera views can provideoblique views, and as appropriate, can be viewed in tiled or stitchedform on either side of the central stereo view in the display. Camerason the tool may also provide video that may be displayed. A central viewof the display can comprise imagery from cameras on the tool or distalcameras and can be configured to switch between sources of imagery forthe central view. In some embodiments, camera location information,camera calibration information, and/or area of interest distance andlocation can be sent to the image processing system. This informationmay be useful for unwrapping, purposefully distorting, electronicallymagnifying or de-magnifying a series of images. Such processing mayenable a multi-camera viewing display to adjust various fixed cameramagnifications and locations within an image. This processing maythereby assist in producing tiled or stitched images where the surgicalsite is substantially centrally located within a field of vision on thedisplay. The processing may also provide an array of views from distalcameras adjacent to a stereo view from the proximal cameras.

Such processing may be employed as well when a user selects one or morecamera views. For example, a plurality of thumbnails or windowspresenting the video feed or icons representing the video feed from aplurality a cameras may be shown on the display, for example, off to theside of the screen. The user may select from these, for example, byclicking on the icon or thumbnail or enlarging the thumbnail or window.The user may also potentially move these more central video feeds and/orarrange these video feeds in some fashion such as in a manner consistentwith the geometric arrangement of the camera or cameras with respect toeach other and/or the retractor or surgical site. In some embodiments,when the user identifies a plurality of icons, thumbnails, and/orwindows the processor automatically arranges the corresponding videofeed in such a geometric arrangement. The user may select videofeeds/images from a subset of the total number of cameras on theretractor and/or tool. One or more images may be used as a backgroundover which other images are shown, e.g., as PIP or in a tiledarrangement. The user can for example identify one streaming videowindow, icon, or thumbnail to be used as the background image. This maybe a wide field of view camera or a surgical microscope view (asdiscussed below). In some embodiments, multiple images are stitchedtogether to obtain this background view. In various embodiments thebackground view is larger than the tiled or PIP image and may includesubstantially the entire view provided by the process such as at least75%, 80%, 90%, 95% or 100% of the screen. In some embodiments, theprocessor selects which image to show as background. The user may forexample identify a plurality of images (e.g., by selecting windows orthumbnails displayed on the display) and the processor may show asdefault one of those images as a background image.

In various embodiments, the display may show a first image or videostream or feed enlarged on the screen, for example, such as at least75%, 80%, 90%, 95% or 100% of the screen. The user may select a secondvideo from another camera by identifying a streaming video window, iconor thumbnail corresponding to the other camera. The user may enlarge theimage from that other camera. The processor may automatically reduce thefirst image or remove the image altogether when the user enlarges thesecond image to a certain size. This threshold size may be for exampleleast 75%, 80%, 90%, 95% or 100% of the screen. The user may set thisthreshold size and/or the processor may have a threshold size for thisfunction. Similarly, the user could select a third video image byidentify another camera and by enlarging the images beyond thethreshold, the user could cause a window corresponding to the secondimage to close or be reduced to a thumbnail. The video stream window,icon, or thumbnail for that second image as well as for the first may beavailable for the user to identify (e.g., by clicking on or enlarging)and to enlarge as a background image. In some embodiments any of theseimages may be used as background for one or more other images, forexample, that are each less than 75%, 80%, 90%, 95% or 100% of thescreen. These smaller images may be shown PIP or tiled in front of theother image which is used as background. A wide variety of othercapabilities may be integrated in the graphic user interface. In variousembodiments, however, the user has the ability to select one or morecameras, represented by icons or thumbnails or windows which show videostreams from the cameras, and enlarge and or reposition the imagesprovide by those camera. The user may, for example enlarge and/orposition those images more prominently and/or more centrally on thedisplay. The user may arrange the images if images from a plurality ofcameras are selected. In some embodiments the processor mayautomatically (at least as a default) position and/or enlarge to aspecific size images that are selected by a user. In some embodiments, auser can select a first video window on a first display and send thefirst video window to a second display. This can make the system swapthe videos such that the video that was previously displayed on thesecond display is displayed on the first display and vice versa. In someembodiments, this functionality is configured to provide aquick-switching functionality to easily and quickly switch between videofeeds. In some embodiments, a mode can be provided wherein by selectingthe first video window on a first display, the first video window issent to a second display; however, the video on the second displayremains the second display together with the first video. The user mayselect which mode by providing different input, for example, clickingversus double clicking, etc.

FIG. 19 illustrates an example configuration of proximal cameras thatprovide a background or main view and a stereo view. FIG. 19 shows thecamera views provided by the camera which may be disposed on theretractor and include a turning mirror such as provided by a prism(e.g., a right angle prism) or other optical element that redirected theimage. Such an optical element may be a reflective surface angled, forexample, at 45° or other angle with respect to the detector array of thecamera. FIG. 4C shows an example of such an optical element configuredto turn the optical path. Other types of optical elements such as prismshaving multiple reflections can be used to redirect or turn the light.FIG. 21 also shows a tubular retractor having an optical designconfigured to turning the optical path downward into the surgical siteand yet provide a low profile. In particular, using such an opticalelement such as a turning mirror or prism or other optical elementconfigured to suitably redirect the optical path, may provide thedesired camera view while reducing the profile of the camera that wouldotherwise potentially introduce obstruction and reduce access forsurgical tools to the surgical site by the retractor.

The background or main view provide by the camera(s) shown in FIG. 19can have a relatively larger field of view compared to the stereo fieldof view and imagery from the cameras can be displayed together (e.g.,tiled, stitched, and/or via picture-in-picture). Any number of proximalwide field of view and/or stereo view cameras can be used to provide adesired image quality and display capabilities.

The image processing system can be configured to receive image data fromproximal and distal cameras having various positions and/ororientations. The image processing module can tile these images fordisplay wherein tiling can include outputting the images withoutstitching them together. The image processing system can tile the imagesbased at least in part on a region being imaged, a field of view of thecamera, camera position, a selection by a user, automated criteria, orany combination of these. A size and/or position of tiled images outputto a display can be based at least in part on a field of view, a regionbeing imaged by the camera, camera position, a relative importance ofthe imagery, a selection by a user, an automated placement, or somecombination of these or other criteria. In some embodiments, the imageprocessing system receives camera position and orientation informationand uses this information to tile images according to regions thecameras are imaging. In some embodiments, the image processing systemprocesses the imagery to magnify, de-magnify, distort, tilt, orotherwise transform the image prior to tiling the images for display. Insome embodiments, the image processing system provides images to thedisplay system which tiles the images on the display according to userselection and/or other criteria. In some embodiments the imageprocessing system provides cues to the view so that the view can readilyascertain the orientation of the cameras. The images may, for examplemay be modified (e.g. distorted) or a symbol (e.g., an arrow) may beincluded to show the direction downward into the surgical site. In someembodiments, video data can be received from three or more cameras. Auser can select at least two of these cameras for simultaneous viewing.The video from the selected cameras can be presented to the user and insome embodiments the image processing system can arrange and/or orientthe video according to physical arrangement of the cameras, fields ofview of the cameras, user configuration settings, or the like. Thenon-selected cameras can be configured to still provide video data tothe system, and the video from the non-selected cameras can be obscured,hidden, dimmed, made transparent, or some similar effect or may belocated non-centrally (e.g., placed moved off to the side or to anotherdisplay) so that the video from the selected cameras is more prominentin the display.

The image processing system can be configured to output stereo imagedata using various methods. In some embodiments, images from one or morecameras designated as “left” cameras can be displayed on a “left”display or on a portion of the display system designated as a “left”portion, which are viewed by the left eye and not the right, forexample, via the left ocular. Similarly, images from one or more camerasdesignated as “right” cameras can be displayed on a “right” display oron a portion of the display system designated as a “right” portion whichare viewed by the right eye and not the left, for example, via the rightocular. Selective viewing of the left and right images by the left andright eye provides the three-dimensional effect. Accordingly, thismethod can provide stereo viewing on the display system for the surgeon.Similarly, as illustrated in FIG. 15, a divided sensor can be used toacquire stereo data. The appropriate imagery can be displayed on aleft-eye display panel and a right-eye display panel. In this way, aright-eye and left-eye view is provided in real-time and the viewerfuses the images in their mind to create a 3-D effect. In someembodiments, the image processing system can process stereo data tooutput 3-D images to be displayed on a single monitor or display capableof presenting 3-D images. Such displays may, for example, modulatebetween the left and right image and selectively modulate the ocularsthrough which the viewer sees the display, coordinating the passage oflight through the left ocular and blocking of the right ocular at thesame time that the display shows the left image. Likewise, the displaymay coordinate the passage of light through the right ocular andblocking of the left ocular at the same time that the display shows theright image.

In some embodiments, stereo data can be overlaid on a background or mainview provided by a relatively wide field-of-view camera or cameras. Thenon-stereo images would be displayed at the same time by both left andright displays or/and at the same time through both left and rightoculars. Accordingly, while the stereo portion of the images would bedifferent for the left and right display or images, the non-stereoportions would be the same.

In some embodiments, the field of view of the stereo camera or camerasis less than total vision width (e.g., about 170 degrees horizontal andabout 110 degrees vertical) and can be less than or equal to about 60degrees, less than or equal to about 50 degrees, less than or equal toabout 30 degrees, between about 30 degrees and about 60 degrees, and/orbetween about 50 degrees and about 55 degrees full width half maximum(FWHM). In some embodiments, the field of view of monocular camerasproviding a wide field of view can be at least about 70 degrees and/orless than about 120 degrees, at least about 90 degrees and/or less thanor equal to about 110 degrees, or at least about 70 degrees and/or lessthan or equal to about 90 degrees FWHM.

The image processing system can be configured to process images from asurgical tool and/or images of the surgical tool to provide outputimagery that enhances or improves a surgeon's situational awareness. Insome embodiments, cameras associated with a surgical tool can moveduring surgery and their position and/or their orientation can betransmitted to the image processing system to enable an output displayof a close-up of tool-tissue interaction. This output display can bestitched with other imagery from other cameras or it can be provided asits own video stream or imagery such as picture-in-picture or viatiling. The output can be, for example, displayed as apicture-in-picture overlaid on a central scene such as the main orbackground view where the picture-in-picture display provides amagnified view of a targeted site. This picture-in-picture view can beenlarged or magnified to provide visual feedback to a surgeon to enhancethe surgeon's ability to manipulate the surgical tool in a desiredfashion. In certain implementations, an orientation of thepicture-in-picture display can remain unchanged throughout the surgery,regardless of any change in orientation of the surgical tool. In suchimplementations, the image processor may reorient the image based oninput from tracking sensors (e.g., IMUs) on the tool so thatpicture-in-picture image does not move as the tool is moved. In certainimplementations, the orientation of the picture-in-picture display doeschange, for example, with rotation of the surgical tool.

In some embodiments, a portion of the surgical tool may be in view ofcameras positioned on the retractor during surgery. To reduce oreliminate obscuration of the surgical site due at least in part to thesurgical tool, the image processing system can process images from thecameras to output imagery of the surgical site having a variablytransparent surgical tool displayed thereon rather than includingimagery of the surgical tool itself. For example, a portion of thesurgical tool within view of the cameras can be rendered based on CAL)models of the tool and 6-degree of freedom tracking information. Byproviding this variably transparent tool representation, a trajectory ofthe surgical tool can be monitored on the display system while the toolis being manipulated. In some embodiments, the image of the tool as seenby cameras can be colored in a partially transparent manner whendisplayed over other views provided by other cameras. The transparencywill permit the other camera views to show through and reduce theobscuration provided by the tool.

In some embodiments, a camera view from a side that is left (right) ofthe surgical tool, from the point of view of the surgeon, can be used toreduce or eliminate tool obscuration for right-handed (left-handed)surgeons. This can be done to reduce tool obscuration which can causeretinal rivalry where the surgeon is right-eye (left-eye) dominant.Accordingly, in some embodiment, the user can select a mode, in thisexample, right hand mode (or left hand mode) and the image processingmodulate may select which images to display based on that mode selectedwithout the user necessarily specifying the cameras and/or images.

FIG. 17 illustrates an example display incorporating image data fromproximal and distal cameras, as in FIG. 16, along with apicture-in-picture view of imagery acquired by a camera associated witha surgical tool. The image information from cameras associated with thesurgical tool can, for example, be displayed overlaid on the main orbackground image provided by proximal wide field of view cameras. Thepicture-in-picture presentation can be in stereo or it can be monocular.In some embodiments, the picture-in-picture display is presented with adifferent magnification or zoom than the background or main view. Insome embodiments, the picture-in-picture display has a level of opacitythat allows the surgeon to view image data displayed underneath thepicture-in-picture.

As discussed above, the cameras can be arranged and positioned toproduce an image of the surgical field having vertical and horizontaldirections the same or substantially the same as the vertical andhorizontal directions that the surgeon associates for the surgicalfield. If the cameras are not positioned correctly, the image of thesurgical field may be rotated on the display such that vertical andhorizontal directions on the display do not correspond to vertical andhorizontal directions that the surgeon associates with the surgicalfield as oriented for the surgical procedure. Incorrect positioning orexcessive rotation (e.g., greater than 30°) of the surgical field withrespect to the vertical and horizontal directions on the display candecouple hand-eye coordination.

Accordingly, in various embodiments, the image processing system candisplay images such that the horizon of the displayed images from thecameras on the retractor remains substantially parallel to the horizonof the acquisition system. In some embodiments, the image processingsystem rotates and/or repositions acquired images when displays suchthat the display horizon is parallel to the acquisition horizon which istypically perpendicular to the gravity vector, or in other words, theacquired images are displayed in an upright orientation relative to thehorizon. As illustrated in FIG. 20A, in some embodiments, the surgicalvisualization system includes a horizon mechanism that rotates tomaintain a substantially consistent orientation relative to gravity suchthat acquisition and display horizons remain substantially parallel withlittle or no image rotation performed by the image processing system. Insome embodiments, the horizon mechanism can include a camera ringassociated with the retractor that can be rotated while viewing asterile alignment and/or white balancing target to address any alignmentissues before performing surgery. In some embodiments, imagery fromcameras associated with a surgical tool can be rotated when the tool isrotated or they can remain horizontal upon tool rotation, as illustratedin FIG. 20A.

In some embodiments, a visual axis line (the line that is perpendicularto both the visual axes of the surgeon's eyes) can be parallel to astereo sensor line (the line perpendicular to both the optical axes ofoptical sensors providing stereo information). For example, if a z-axisis defined as an axis that generally runs along a spine of a standingsurgeon (e.g., the z-axis is parallel to the gravity vector), the visualaxis line and the stereo sensor line can be configured to be parallellines when they are projected onto a plane perpendicular to the z-axis.This allows the visual axis line, the stereo sensor line, or both torotate about the x-axis, the y-axis, or both (in a Cartesian coordinatesystem) while maintaining a desired parallel relationship. This parallelrelationship can be useful to reduce disorientation of the surgeon whenviewing imagery on a display where the imagery is displayed such thatthe z-axis of the surgeon and the z-axis of the displayed imagery areparallel. In some embodiments, as illustrated in FIG. 20B, the sensorscan be oriented on retractor blades in a parallel fashion (in FIG. 20B,gravity would be a vector pointing into the page). The sensors on thetop and bottom blades can be configured to provide monocular or stereoimagery, and the sensors on the side blades can be configured to providemonocular imagery. In some embodiments, the top and/or bottom blades aswell as at least one of the side blades can be configured to providestereo imagery. In various of these embodiments, the left and rightcamera apertures for the side blade(s) are arranged to provideconsistent stereo perspective as provided by top and/or bottom blades.Accordingly, as shown in FIG. 20B, the pair of left and right cameraapertures may be aligned in a different direction with respect to thelength of the retractor blades for the side retractors as opposed to forthe top and bottom retractors. Similarly, the left and right cameraapertures are aligned in a different direction with respect to thecentral axis into the surgical site (into the figure) as defined by theretractor as compared to the orientation of the left and right cameraapertures for the top and bottom retractors. In particular for the top,bottom and sides, the left and right camera apertures are arranged alongthe horizontal direction to provide a consistent stereo perspective foreach of the retractors. The result, however, is that the cameras areoriented differently on the top and bottom retractor blades as comparedto the side retractor blades.

This configuration for the top and bottom retractors can be advantageouswhen it is desirable to conserve space and provide more space for toolsin the pathway to the surgical site provided by the retractor. Foldedstereo top and bottom cameras, as shown in FIG. 20B, provide arelatively low profile. Stereo pairs on side cameras cannot typically bemade small due at least in part to the horizontal axis being 90 degreesrelative to the top and bottom configuration. In some embodiments, sidecameras could be made relatively small, if there are no prisms forfolding or deviating a line of sight, which can be useful in the 90degree or opposing view configuration.

In various embodiments, the sensors can be oriented such that theiroptical axes are normal to gravity, parallel to gravity, or have anotherinclination or declination relative to gravity, as described herein. Insome embodiments, the display system can be movable (e.g., using anarticulating arm) to allow the surgeon to move relative to the patient,as described herein. The display system can be configured to have motionsensors such that this movement is detected and an alert or warning isprovided indicating to the surgeon or other operator that the retractorshould be repositioned to maintain a desirable parallel orientationbetween the surgeon, the display, and the cameras.

In some embodiments, the image processing system can provide a default,predetermined, or appropriate image rotation and/or position based atleast partly on a position and/or field of view of the camera which isproviding imagery. Comparing FIGS. 20C and 20D, cameras 1-4 provideimagery of an object to a display system. In FIG. 20C, the imageprocessing system tiles the images from the four cameras in locations onthe display corresponding to their locations on the retractor or fieldof views. The locations on the display can be based at least partly onthe positions or field of views of the cameras relative to the object,surgical field or site, to the surgeon, or to another reference framethat is automatically or manually selected. In some embodiments, camerasat opposing positions can be configured to provide stereo video data.For example, Camera 1 and Camera 2 can include cameras configured toacquire stereo data. Additionally, in some embodiments, Camera 1 andCamera 2 can be rotated relative to one another, for example, by 180degrees. In some embodiments, Camera 3 and Camera 4 can be configured toprovide panoramic and/or monocular video. Similar to Cameras 1 and 2,Cameras 3 and 4 can be rotated with respect to one another. In FIG. 20D,the image processing system positions the tiled images on the display asin FIG. 20C, but the image processing system also performs a rotation onthe imagery (e.g. rotation of the separate images). The rotation can beconfigured to provide a uniform direction for the displayed object, asillustrated in FIG. 20D. For example, imagery from camera 1 can berotated 180 degrees, imagery from camera 2 can be presented withoutrotation, camera 3 can be rotated 90 degrees, and camera 4 can berotated 270 degrees. As a result, the object presented in the displaywill have the same relative orientation. This can facilitate theunderstanding of the observer as the object has the same orientation inthe display even though it is viewed from different viewpoints. In otherembodiments not all the images are reoriented. In various embodiments,for example, the images from the video images from camera 2 are rotated180° with respect to camera 1, or vice versa while the images fromcamera 3 and 4 are not rotated. In other embodiments, the video imagesfrom camera 2 are rotated 180° with respect to camera 1, and the videoimages of camera 3 is rotated by 90° with respect to camera 4 or viceversa. Other variations are possible. Generally, maintaining camera 2unrotated may increase the situational awareness or reduce confusionwhen using the surgical visualization system as camera 2's horizon mayclosely align with the horizon of the operator. In some embodiments,camera 2 can provide video data that is enlarged and displayed as abackground image for the other three tiled videos from the othercameras. These concepts may apply to more or less number of tile cameraviews.

In some embodiments, the image processing system calculates the positionand/or rotation for the displayed images based at least partly ontracking data, information provide with the camera such as memory(EEPROM) storing information regarding that camera, user selection, or acombinations thereof. In some embodiments, the image processing systemcan perform a default or predetermined rotation and/or positioning ofthe displayed image based on configuration settings or a configurationof the retractor cameras (e.g. positions and/of field-of-views). In someembodiments, the user can select the position and/or rotation of thetiled images using video windows, icons or thumbnails presented on agraphical user interface. The thumbnails can be rotated reduced-sizeimages that correspond to the video being provided by the retractorcameras. The video windows, icons or thumbnails can remain unrotated, insome implementations, for example, to provide cues as to perspective ofthe cameras or can be rotated. The rotation of the video in thisfashion, as illustrated in FIG. 20D, can be applied to monocular orstereo imagery. As described above, in various embodiments, the imageprocessor can arrange the videos from the cameras selected by the userfor example in an arrangement consistent with the geometricalarrangement of the cameras or their fields of view. The user can, forexample, select images or video feeds from cameras by selecting videowindows, icons or thumbnails. The processor can automatically enlargeand reposition the videos from these cameras, for example, to improvethe ability of the surgeon to see the detail in the video. In someembodiments the processor moves the video to a more central position onthe display or to a desired position on the display. In someembodiments, the user enlarges the video manually and/or repositions thevideo manually, for example, by enlarging the video window, thumbnail oricon and/or repositioning it. Again, the user may enlarge the video toimprove ability to see detail in the video and may move the video in amore central position or may move the video to a desired location. Insome embodiment, reduced size video windows, thumbnails or icons are ona first touch screen display used as a graphic user interface for thenurse or technician and possibly the surgeon, and the enlargerepositioned images are on a second display, such as the binoculardisplay viewed extensively by the surgeon throughout the procedure. Thevideo streams can thus switch from one display to another in variousembodiments depending on the surgeons interested in viewing the displaywhile for example performing a detailed examination of the surgicalsite, performing tool manipulations in the surgical site, acting uponthe surgical site, etc.

In some embodiments the group of videos can be rotated as one. Forexample, the processor provides that a plurality of videos such as thosetiled videos shown in FIGS. 20C and 20D may be rotated as one. In someembodiments, the user initiates and/or controls the rotation of thevideos as a group, although the processing system may do so as well. Insome embodiments, the processing system automatically rotates one ormore of the individual videos in the group of tiled videos when thegroup as a whole is rotated.

As shown in FIGS. 20C and 20D, the tiled videos can be configured to bedisplayed on a background videos that is provided by another camerasystem, a mosaic of videos, auxiliary cameras, cameras on a surgicaltool, a static image, or the like. In various embodiments, as shown inFIGS. 20C and 20D, tiled videos are spread out, for example, presentedmore toward the a periphery of the display or in a manner such that thea vacancy is provided in the center of plurality of images and/or of thedisplay. This can allow, for example, for a presentation of an enlargedpicture-in-picture videos to be displayed (e.g., a tool image) and/ormay be consistent with the concept of a retractor providing an opencentral pathway providing access to the surgical site. In someembodiments, the surgical tool image can be displayed with the tiledvideos from retractor cams and/or with video provided by an auxiliarycamera that provides a surgical microscope view, as described herein. Insome embodiments, the surgical tool videos can be displayed in a centralregion overlaid on the tile videos and can have a relatively fixedorientation or the orientation of the video can be adjusted with videosin the tool's orientation. For example, the surgical tool video can bepresented on the display without undergoing any rotation with the imageprocessing system. In some embodiments, the surgical tool camera maypresent the tool in a fixed position relative to the field of view ofthe surgical tool camera which will provide orientation cues.

Display

Embodiments described herein can provide a display having a form factorand articulated arm as well as binocularity that is familiar to surgicalmicroscope users without the associated disadvantages. In contrast tothe typical surgical microscope apparatus or endoscope, however, variousembodiments described herein can avoid the need for operator control ofthe optical focus, positioning, distance to target, etc. Embodimentsdescribed herein can, for example, provide visualization that isgenerally always in focus, oriented, and provides a view of virtuallythe entire surgical field, emphasizing current task space as desired.Embodiments described herein may also avoid the glare/ambient light andview angle problems associated with looking at conventional endoscopeflat panel, TV-like displays. Embodiments described herein may alsoprovide a display that is near the surgical site and near the surgeonand oriented to be convenient for the surgeon to view the display whileperforming surgery, e.g., without having to turn substantially away fromthe surgical site to view the display. Embodiments described herein canalso avoid the vertigo/nausea and weight/bulk on a user's head, problemswhich are typical with head-mounted displays.

Additionally, embodiments described herein can provide displays havingadvantages over current robotic-assisted surgery systems, such as theDaVinci. In contrast to the DaVinci approach, embodiments describedherein provide a display at the patient, rather than at a remotelocation across the room. As noted above, embodiments described hereinmay provide a display that is similar to the operating microscope towhich surgeons are familiar and comfortable but more compact andlighter. By providing a display at the patient site, the user is able tovisualize body, wound, respiratory movement, endotracheal tube security,IV tubing, central lines, EKG leads, sterility/draping, and otherrelevant features easily throughout the surgical procedure. In contrastto an operating microscope where many optical elements are aligned fromobjective to ocular in a conventional manner, the display can be of amore compact form, allowing the surgeon to look under or over thedisplay. Additionally, due to the electronic nature it can be folded inan advantageous manner to allow more working space for the operator'shands and tools.

The composite image described above can be displayed in a variety ofways. For example, a flat screen, curved (e.g. hemispherical) screen, orprojection directly into a user's eyes or glasses may be used. In thecase of a flat screen, the display can be, for example, two OrtusTechnology, 4.8-inch color LCDs, arranged in landscape orientation witha resolution of 1920×1080, RGB 458 ppi, viewing angle of 160 degrees(horizontal/vertical), color depth of 16.77 million colors, NTSC 72%color gamut, with LED backlight. Different displays in respective leftand right optical paths to the eyes can provide stereo and 3-D. Ahalf-silvered mirror beam splitter or multiple mirrors or reflectivesurfaces (e.g. Wheatstone configuration) can be used to combine imagesfrom the displays or portions thereof and/or arrange them adjacent toeach other for left and right eye viewing.

In some embodiments, an array of multiple, side-by-side emissive OLEDdisplays in Wheatstone or over/under configuration may be used. In someembodiments, a LCoS pico projector array with rear screen or frontscreen projection may be used to achieve sufficient field of view(“FOV”) and pixel density. Another option is to employ an array of OLEDsprojecting on a 3M rear screen. OLED pico projectors have limitedbrightness, but in configurations in which the display is set up similarto a microscope as discussed below, where little or no ambient light isintroduced, the limited brightness presents less of an obstacle. A widevariety of configurations are possible.

As referred to above, the display can be enclosed to eliminate straylight, for example in a manner similar to a standard microscope viewingplatform. In particular, the use of a microscope-like display can avoidproblems associated with insufficient brightness, as the oculars mayblock out ambient light.

As illustrated in FIGS. 1 and 21B, the surgical visualization system 1can include a viewing platform 9 with oculars 11 for the surgeon to useto view a display of videos acquired with the various cameras in thesurgical visualization system. FIG. 21B illustrates an embodiment of thesurgical visualization system 1 having an articulating arm 7 b for animaging system 18 that can be configured to provide video similar to adirect-view surgery microscope. The imaging system 18 can be configured,then, to provide a surgical imaging system configured to provide anelectronic microscope-like view that can comprise video of the work siteor operational site from a position above the site (e.g., about 15-45 cmabove the surgical cite) or from another desired angle. By decouplingthe imagers 18 from the display, the surgeon can manipulate the surgicalimaging system to provide a desired or selected viewpoint without havingto adjust the viewing oculars. This can advantageously provide anincreased level of comfort, capability, and consistency to the surgeoncompared to traditional direct-view operating microscope systems. Insome embodiments, as described herein, the imagers 18 can be located onthe viewing platform 9, on a dedicated articulating arm 7 b, on adisplay arm 5, or detached from other systems. The imagers 18 cancomprise a camera configured to be adjustable to provide varying levelsof magnification, viewing angles, monocular or stereo imagery,convergence angles, working distance, or any combination of these.

The viewing platform 9 can be equipped with wide field-of-view oculars11 that are adjustable for refractive error and presbyopia. In someembodiments, the oculars 11, or eyepieces, may additionally includepolarizers in order to provide for stereoscopic vision. The display canbe supported by an articulated arm 7 or 7 b, such that it may bepositioned for the user to comfortably view the display while inposition to perform surgery.

In some embodiments, the image processing system and the display systemare configured to display imagery placed roughly at infinity to reduceor eliminate accommodation and/or convergence when viewing the display.An optical display system comprising a pair of oculars, objectives, anda display resembling a binocular microscope can be employed. The displaydevices such as liquid crystal displays can be imaged with the objectiveand the pair of oculars and imaging optics within the display. Theobjective and the oculars and imaging optics within the display can beconfigured to produce an image of the displays at infinity. Sucharrangements may potentially reduce the amount of accommodation by thesurgeon. The oculars can also have adjustments (e.g., of focus or power)to address myopia or hyperopia of the surgeon. Accordingly, the surgeonor other users may view the displays through the oculars without wearingglasses even if ordinarily prescription glasses were worn for otheractivities.

In some embodiments, the viewing platform can include one or moreimagers configured to provide electronic microscope-like imagingcapabilities. FIG. 21C illustrates an example surgical imaging system 51attached to an articulating arm 7, the system 51 including one or morecameras 18 mounted on a viewing platform 9. The cameras 18 can beconfigured to provide imagery of a worksite. The image data can bepresented on a display that the user can view using oculars 11 mountedon the viewing platform 9. This design can be used to mimic otherdirect-view microscopes, but it can also be configured to provideadditional capabilities. For example, the surgical imaging system 51 canbe configured to have a variable working distance without adjusting theviewing platform 9 or the articulating arm 7. The surgical imagingsystem 51 can be configured to provide image processing capabilitiessuch as electronic zooming and/or magnification, image rotation, imageenhancement, stereoscopic imagery, and the like. Furthermore, theimagery from the cameras 18 can be combined with imagery from retractorcameras, from surgical tool cameras, and the like as described ingreater detail herein.

In some embodiments, for example, the displays provide video windows,icons and/or thumbnails to identify cameras 18 that provide a surgicalmicroscope view as well as cameras mounted on the retractor. The usermay for example select to display either the surgical microscope view orone or more views from the retractor. The user interface may provide aneasy way to switch back and forth that may be easier than moving videowindows or icons. An icon, button, or other graphic may enable swiftswitching by the user back and forth between the two types of views.Such flipping back an forth may be useful at the early stage of theprocedure when the incision is being made and the retractor is beingintroduces as well as when tools are being introduce into the surgicalsite. Flipping back and forth may similarly be useful at the end of thesurgical procedure with extraction of items from the surgical site.However, switching back and forth may also be useful in the middle ofthe procedure.

In some embodiments, both the surgical microscope view as well as theretractor camera views may be simultaneously shown. For example, thesurgical microscope view may be used as a large wide field of viewbackground and one or more retractor camera views may be use as theforeground for example in PIP or tiled format.

Although the discussion and drawings such as FIGS. 1 and 21B considerimages from retractors, numerous embodiments may involve at least oneauxiliary camera 18 and one or more other cameras that are not disposedon retractors but are disposed on other medical devices. These medicaldevices may include devices introduced into the body such as endoscopes,laparoscopes, arthroscopes, etc. Accordingly, one or more displays suchas the at least one display included in the viewing platform 9 may beused to provide a surgical microscope view using one or more camerassuch as the auxiliary camera(s) 18 as well as to display views from oneor more cameras located on such medical devices other than retractors.In some embodiments, cameras from a variety of sources, e.g.,retractors, surgical tools, and other medical devices, in anycombination, may be viewed on the display(s) on the surgical platformtogether with the surgical microscope view from the auxiliary cameras18. As discussed above, various embodiments provide the ability toswitch between the surgical microscope views and the views provide bycameras associated with other medical devices. Alternatively, the imagesfrom the auxiliary camera(s) 18 can be viewed simultaneously with cameraview provided by cameras disposed on other medical devices. In some suchembodiments, the surgical microscope view may be displayed as a widefield of view background view or main view with other views in theforeground or tiled thereon. As described herein, the displays mayprovide 3D thus any of the images and graphics may be provided in 3D.

In various embodiments a virtual touchscreen may be provided by theauxiliary cameras 18 or other virtual touchscreen cameras mounted to theviewing platform 9. Accordingly, in some embodiments a user may providea gesture in the field of view of the auxiliary cameras and/or virtualtouchscreen cameras and the processing module can be configured torecognize the gesture as an input. In some embodiments, this camera viewcan be simultaneously displayed in conjunction with icons, buttons,thumbnails, or other graphics for which gestures may be coordinated topermit the user to provide input. For example, the display(s) in theviewing platform 9 may provide an icon or thumbnail as well as viewsfrom the auxiliary camera 18 which show hand gestures or gestures madewith a surgical tool held by a surgeon. The surgeon may for exampleidentify a video window, thumbnail, or icon with a gesture of a surgicaltool and activate the view of a particular camera such that the view ofthat camera is enlarged on the screen. In some embodiments, a graphicrepresentation of the surgeon's hand is shown in the display when thesurgeon's hand is in the field of view of the virtual touchscreencamera(s). In some embodiments, the virtual touchscreen can mimic amulti-touch display, allowing the surgeon to manipulate objects in thevirtual environment with their hands and/or fingers. Although thevirtual display has been described in the context of the auxiliarycameras 18, other cameras, e.g., virtual reality input cameras, possiblyin addition to the auxiliary cameras 18 may be used. These cameras maybe disposed on the viewing platform 9 or elsewhere, such as theadditional articulated arm 7 b shown in FIG. 21B. In some embodiments, auser may provide a voice command and the processing module can beconfigured to recognize the voice command as an input. Providing thevirtual touch screen can help to create an immersive display experiencefor a user as it may not be necessary for the user to look away from thedisplay to accomplish a majority of the tasks to be done during asurgical procedure.

As described herein the displays may provide 3D thus the virtual realityinterface may appear in 3D. This may increase the immersive quality ofthe viewing experience, enhancing the detail and/or realisticpresentation of video information on the display.

In some embodiments, as illustrated in FIG. 21D, the surgical imagingsystem 51 includes an isocenter positioning system 52 attached to theviewing platform 9. The isocenter positioning system 52 can include asingle track or guide configured to move and orient the cameras 18 suchthat they are substantially pointed at a single point 53, the isocenter.In some embodiments, a second track or guide can be attached to thefirst guide in an orthogonal manner to provide movement along 2dimensions while substantially maintaining the pointing angle towardsthe isocenter 53. Other configurations can be used to provide isocenterpointing capabilities, such as articulating arms, electro-mechanicalelements, curved friction plates, etc. In some embodiments, asillustrated in FIG. 21D-2, the imaging system is configured to move inan isocenter manner. This can be used to enhance dexterity of the userof the system because hand-eye coordination is increased or maximized.Such enhanced dexterity can be vital for prolonged and/or difficultsurgery. In the displayed embodiment, the horizons of the acquisitionsystems are configured to be horizontal to match the horizon of thedisplay system and the user. As shown in FIG. 21D-2, in variousembodiments, a stereo imaging system may be maintained in a horizontalconfiguration as it is moved across a range of locations to avoidconfusion for the user viewing the video from the stereo camera. Bymaintaining a common relative horizon between the display and theacquisition system, the user can relatively easily translate hand motionto manipulation of objects in the display, which may not be the casewhere translation of the acquisition is accompanied by a relativerotation between the display and the acquisition system.

In the embodiments illustrated in FIGS. 21D and 21D-2, the isocenterassemblies can be a part of the display system or a separate,independent system. For example, the viewing platform 9 can be mountedon a separate articulated arm from the cameras 18. Thus, the display andthe image acquisition of the surgical imaging system can be decoupled,similar to the embodiment illustrated in FIG. 21B. By decoupling theisocenter cameras 18 from the display ergonomic benefits are providedsuch as, for example, the surgeon does not need to be looking throughbinoculars for an extended period of time or at an uncomfortableposition or angle. In various embodiments, a common relative horizon forboth the display and the acquisition system is also employed to avoidconfusion for the user viewing the video from the stereo camera asstated above.

In some embodiments, the distance between the surgical site of interestand the imagers, e.g., the working distance, can be at least about 20 cmand/or less than or equal to about 40 cm, at least about 10 cm and/orless than or equal to about 50 cm, or at least about 5 cm and/or lessthan or equal to about 1 m.

The user can interact with the surgical imaging system 51 to select aworking distance, which can be fixed throughout the procedure or whichcan be adjusted at any point in time. Changing the working distance canbe accomplished using elements on a user interface, such as thegraphical user interface described herein with reference to FIGS. 24 and24B, or using physical elements such as rotatable rings, knobs, pedals,levers, buttons, etc. In some embodiments, the working distance isselected by the system based at least in part on the cables and/ortubing being used in the surgical visualization system. For example, thecables and/or tubing can include an RFID chip or an EEPROM or othermemory storage that is configured to communicate information to thesurgical imaging system 51 about the kind of procedure to be performed.For an ENT/Head/Neck procedure, the typical working distance can be setto about 40 cm. In some embodiments, the user's past preferences areremembered and used, at least in part, to select a working distance.

In some embodiments, gross focus adjustment can be accomplished manuallyby positioning the cameras 18 and arm 7. The fine focus adjustment canbe done using other physical elements, such as a fine focusing ring, orit can be accomplished electronically.

In some embodiments, the magnification of the surgical imaging system 51can be selected by the user using physical or virtual user interfaceelements. The magnification can change and can range between about 1×and about 6×, between about 1× and about 4×, or between about 1× andabout 2.5×. Embodiments may be able to change between any of these suchas between 2.5× and 6× or between 2.5× and 6×. Values outside theseranges are also possible. For example, the system 51 can be configuredto provide magnification and demagnification and image inversion, with arange from about −2× to about 10×, from about −2× to about 8×, fromabout −2× to about 4×, from about −0.5× to about 4×, or from about −0.5×to about 10×. The surgical imaging system 51 can be configured todecouple zoom features and focus adjustments, to overcome problems withtraditional operating room microscopes. In some embodiments, thesurgical visualization system 51 can be used to provide surgicalmicroscope views. In some embodiments, the surgical imaging system 51can decouple instrument myopia by providing an electronic displayinstead of a direct view of a scene. The electronic displays can beconfigured to be focused at varying levels of magnification allowing theuser to view the displays without adjusting the oculars betweenmagnification adjustments. Moreover, in various embodiments, the ocularscan be configured to provide continuous views at infinity. In someembodiments, however, the principal user of the surgical imaging systemmay select an accommodation level for the oculars, rather than using arelaxed view provide by the electronic displays. The electronicdisplays, in various embodiments, however, can remain in focus and theocular adjustments do not affect the focus of the various videoacquisition systems. Thus, adjustments by the principal user do notaffect the views of the other users of the system viewing, for example,other displays showing the video, as the cameras/acquisition systems canremain focused. In some embodiments, the surgical imaging system 51 canbe focused at a relatively close working distance (e.g., a distance witha relatively narrow depth of field) such that the image remains focusedwhen moving to larger working distances (e.g., distances with broaderdepth of field). Thus, the surgical imaging system 51 can be focusedover an entire working range, reducing or eliminating the need torefocus the system after magnification or zoom adjustments are made.

FIGS. 21E and 21F illustrate an embodiment of the surgical imagingsystem 51 having an optical system 53 mounted under the viewing platform9. As illustrated, the optical components are shown as free-standing toshow the structure of the components, but in practice the opticalcomponents 53 will be mounted within or on a structure attached to theviewing platform. In some embodiments, the optical system 53 and/or thecameras 18 (discussed above) can be modular and can be selected andswapped for use with the surgical imaging system 51.

The optical system 53 is configured to provide stereo image data to theimaging system 51. The optical system 53 includes a turning prism 54 tofold the optical path underneath the viewing platform 9 to decrease thephysical extent (e.g., length) of the imaging system under the viewingplatform 9.

In some embodiments, the optical system 53 comprises a Greenough-stylesystem wherein the optical paths for each eye have separate opticalcomponents. In some embodiments, the optical system 53 comprises aGalilean-style system wherein the optical paths for each eye passthrough a common objective. The Greenough-style system may be preferablewhere imaging sensors are being used to capture and convey the imagedata as compared to the Galilean-style system. The Galilean system canintroduce aberrations into the imagery by virtue of the rays for eacheye's optical path passing through a periphery of the objective lens.This does not happen in the Greenough-style system as each optical pathhas its own optics. In addition, the Galilean system can be moreexpensive as the objective used can be relatively expensive based atleast in part on the desired optical quality of the lens and its size.

As shown in FIGS. 21E and 21F, the optical system 53 can include tworight-angle prisms 54, two zoom systems 55, and two image sensors 55.This folding is different from a traditional operating room microscopebecause the optical path leads to image sensors rather than to adirect-view optical system.

In some embodiments, the optical system 53 can have a relativelyconstant F-number. This can be accomplished, for example, by varying thefocal length and/or aperture of the system based on working distanceand/or magnification. In one embodiment, as the focal length changes,the eye paths can move laterally apart (or together), the prisms 54 canrotate to provide an appropriate convergence angle, and the aperturescan change their diameters to maintain the ratio of the focal length tothe diameter a relatively constant value. This can produce a relativelyconstant brightness at the image sensor 56, which can result in arelatively constant brightness being displayed to the user. This can beadvantageous in systems, such as the surgical visualization systemsdescribed herein, where multiple cameras are being used and changing anillumination to compensate for changes in focal length, magnification,working distance, and/or aperture can adversely affect imagery acquiredwith other cameras in the system. In some embodiments, the illuminationcan change to compensate for changes in the focal length and/or theaperture so as to provide a relatively constant brightness at the imagesensors 56.

The optical assembly 53 can include a zoom system 55 configured toprovide a variable focal distance and/or zoom capabilities. AGalilean-style stereoscopic system generally includes a common objectivefor the two eye paths. When this optical system is imaged with imagesensors 56, it can create aberrations, wedge effects, etc. that can bedifficult to compensate for. In some embodiments, the surgical imagingsystem 51 can include a Galilean-style optical system configured tore-center at least one of the stereo paths to a central location throughthe objective lens, which can be advantageous in some applications.

In some embodiments, the real-time visualization system utilizes aGreenough-style system. This can have separate optical components foreach stereo path. The optical assembly 53 can be configured to providevariable magnification and/or afocal zoom and can be configured tooperate in a magnification range from about 1× to about 6×, or fromabout 1× to about 4×, or from about 1× to about 2.5×.

The distal most portion of the Greenough assembly 53 can be similar infunctionality to an objective lens of a typical, direct-view operatingroom microscope with the working distance set approximately to that ofthe focal length. The working distance, and in some implementations thefocal length, can be between about 20 cm and about 40 cm, for example.In some embodiments the work distance may be adjustable from 15 cm to 40cm or to 45 cm. Other values outside these ranges are also possible. Insome embodiments, the surgical imaging system 51 includes anopto-mechanical focus element configured to vary the focal length of apart of the optical assembly 53 or the whole optical assembly 53.

FIGS. 21G-21K illustrate embodiments of optical assemblies 53 for use ina stereoscopic surgical imaging system, such as those described hereinwith reference to FIGS. 21E-F. FIG. 21G illustrates a side view of anexample optical assembly 53 configured to use a turning prism 54 to foldan optical path from a tissue 57 to a sensor 56 along a lens train 55that is situated near or adjacent to a viewing platform 9. This canadvantageously provide a relatively long optical path in a relativelycompact distance.

FIG. 21H illustrates a front view of an embodiment of an opticalassembly configured to change a convergence angle in a stereoscopicimaging system. The prisms 54 can be the turning prism 54 illustrated inFIG. 21G. The prisms 54 can be configured to rotate to change aconvergence angle, and as a result, a convergence point and/or a workingdistance. The working distance, which can be a distance from the prisms54 to the target 57 (e.g., tissue) can be user-selectable or adjustable.In various embodiments, with increased working distance to the target57, the convergence angle can decrease. Conversely, when the workingdistance gets smaller, the convergence angle can increase (e.g., θ1>θ2).This can be advantageous where the lens path 55 is fixed and the workingdistance is adjustable. The stereo imagery can then be viewed on thedisplay 59 by a user.

FIG. 21I illustrates a front view of an embodiment of an opticalassembly 53 that is configured to maintain a substantially constantconvergence angle. The optical assembly 53 can include two prisms 54 aand 54 b for each optical path, wherein the prisms 54 a, 54 b can moveand/or rotate. For example, when the working distance decreases thefirst set of prisms 54 a can rotate towards one another to decrease aneffective distance between the second set of prisms 54 b. The second setof prisms 54 b can, in turn, rotate to compensate for the changed angleso as to converge on the common target. The second set of prisms 54 bcan direct the light to the first set of prisms 54 a which can thendirect the light down the fixed lens paths 55 (e.g., fixed in theirposition relative to the viewfinder). By providing a relatively fixedconvergence angle, a change in working distance may not requirerefocusing for the user. Maintaining a constant convergence angle,especially a comfortable angle, may reduce the strain on the user suchas a surgeon performing a prolonged, arduous procedure.

FIG. 21J illustrates a front view of an embodiment of an opticalassembly 53 configured to provide a substantially narrow convergenceangle to be able to view stereoscopic imagery through a narrow insertiontube 60 (e.g., a tube partially inserted into a body during aprocedure). A similar assembly 53 can be used as described withreference to FIG. 21I, and the convergence angle can be maintainedsubstantially constant or at least sufficiently narrow to view throughthe insertion tube 60.

FIG. 21K illustrates a front view of an embodiment of an opticalassembly 53 configured to provide a substantially constant convergenceangle by moving the lens paths 55 laterally, e.g., toward or away fromone another. The prisms 54 can be made to have a substantially constantorientation (e.g., no rotation for changing working distances) andcompensation for changing working distance can be accomplished bytranslating the optical paths laterally to separate or join the opticalpaths. The translation of the optical paths can be accomplished usingany suitable means including, for example, electro-mechanical actuators,slides, articulating arms, etc. This can simplify the optical assemblycompared to the embodiments having two sets of prisms as only one set ofprisms may be used when the lens paths are configured to move.

The embodiments of the optical assembly 53 which are configured tomaintain a sufficiently narrow convergence angle can be advantageous asthey allow stereo access to narrow surgical entries by allowing theangle to decrease and avoid clipping one of the stereo paths. Forexample, the left and right lens paths can move closer to one anotherand the prisms can adjust to the proper convergence angle for thatdistance. As another example, the left and right lens paths can remainfixed and there can be sets of prisms for each path configured to directthe light along the lens paths while maintaining a substantiallyconstant convergence angle. In some embodiments, maintaining a constantconvergence angle can be visually helpful to the user when zoom changes,e.g., because the changing depth cues do not confuse the user's eyeand/or brain. In addition, constant convergence may induce less stresson the user.

In some embodiments, the surgical imaging systems 51 can include a fiberoptic light guide (e.g. fiber optic bundle) that connects to the system51 and that can provide illumination to the work site. In someembodiments, integrated into the body of the system 51 can be a mixingrod to reduce or eliminate imaging of the pixilated end of the fibercable onto the surgical site. In some embodiments, the area illuminatedby the system can be configured to match that of the imaging area.

In some embodiments, the optical assembly 53 can be sterilized byautoclave where the assembly can be made to detach from the viewingplatform. In some embodiments, Hall effect switches or sensors can beused for focus and working distance adjustments.

The surgical imaging systems 51 can be configured to provide dual zooms,where a first zoom level is for working distance and a second zoom levelis for magnification to match what a traditional operating roommicroscope provides. The dual zooms can be provided by the systems 51where the optical assemblies 53 include, for example, a sensor, a camerablock, a collimating block made up of a telephoto-like assembly oflenses (e.g., 5 or 6 lenses) with a zoom in front or an afocal changer(e.g., configured to adjust the magnification of each lens path'slenses), a second zoom or afocal assembly (e.g., configured to adjustthe working distance), and a prism assembly that deviates the line ofsight 90 degrees. Accordingly, various embodiments the surgical imagingsystem 51 includes one or more cameras that provide both variable workdistance as well as a least partially independently variablemagnification.

In some embodiments, the display can be a curved surface, for exampleeither projection display or recent generation of flexible LCD or OLEDdisplays having high-resolution (e.g., in excess of 300 ppi). A curveddisplay may provide two advantages: the imaging optics for the displaycan be less complex than for flat panels, and the cone or numericalaperture of each picture element in the display can be directed towardsthe viewing optics and in the periphery of the display, therebyproviding a brighter image less subject to vignetting.

In some embodiments, the display can be a volumetric display comprisingtwo or more transmissive display panels having a single backlightwherein the transmissive display panels are stacked to provide differentplanes of focus for a surgeon. The transmissive displays can be activematrix liquid crystal displays (“AMLCD”) or other types of transmissivedisplays. The backlight can be a fluorescent lamp, LEDs, or othersuitable light source. By having displays positioned in different focalplanes, image data from different focal planes may be presented to thesurgeon with relatively less image processing and/or compressioncompared to a system which combines data from multiple focal planes intoa single image. In some embodiments, a number of cameras can bepositioned at varying depths or having varying focal distances such thatthe displays at different focal planes are configured to display imagedata from cameras positioned or focused at different depths to create adisplay that assists the surgeon in identifying positions of featureswithin displayed images.

The display can show, as an overlay, pre-operative CT, MR, or other 3Dimage datasets from, for example, conventional surgical navigationsystems (e.g., the Medtronic StealthStation or Treon, Stryker SurgicalNavigation System, or Brainlab, among others). In various embodiments,in addition to images, the display can additionally provide numericaldata and/or text. For example, in various embodiments, the display canoverlay information such as distance or tool measurements, transparenttool renderings, camera identification information (e.g., the portion ofthe composite image attributable to a specific optical sensor maygenerate an identifying border around that portion), up/downorientation, elapsed time, and/or one or more still images captured fromone or more optical sensors from a previous time in the operation. Thetracking system can provide 5-DOF (degrees of freedom) or 6-DOF positionand orientation information to conventional surgical navigation systems.Other information, graphic, alpha numeric, or otherwise, can beprovided.

The tool image can be magnified with respect to the wide-field viewimage, and change in image scaling will occur as the tool is moved inand out. In some embodiments, a visual metaphor for embodiments of thedisplay is that of a hand-held magnifying glass for inspecting and doingwork on a smaller region of a larger workpiece, while seeing the largerworkpiece with lower magnification (if any) in more peripheral regionsof the visual field to provide situational awareness. Tool images, forexample, can be superimposed on the background image thereby blockingthat portion of the background image. In various embodiments, the toolimages may be stereo.

FIGS. 22A and 22B show examples of displaying a composite image 600 bystitching and tiling images 602 a-c. When tiled, the individual images602 a-c may each have a readily discernable border that is displayedagainst a background scene or color.

In the case in which the optical sensors 604 a-c are pointed downwardsor upwards within the surgical device, e.g., at an oblique angle, (forexample, the sensors do not form an orthogonal ring of sensors whosechief lines of sight angles are parallel), the images may be rendered astrapezoids. For example, a keystone effect is shown in FIG. 22B whenoptical sensors 604 a-c are pointed down with an angle, θ, relative to asurface of an object 606. The composite image, 600, can comprisetrapezoidal images 602 a-c tiled together without correction for thekeystone effect. Trapezoids are powerful monocular clues for direction.The result may reveal the plurality of optical sensors is downwards orupwards, for example. In some embodiments, discontinuities in the tiledimage 600 can provide visual cues to the operator to provide additionalsituational awareness. Other types of direction markings or indicatorsmay be employed.

In some embodiments, it is possible for the user to reposition a singleoptical sensor or optical sensor pair from the array or a modularclip-on optical sensor in such an orientation that the image could notbe stitched into the composite image. This image could, in someembodiments, be displayed as a picture-in-picture (PIP) overlaid in aperipheral portion of the composite image in the appropriate location.This PIP can be provided with a clear border so as to indicate that theimage is not stitched into the composite image, but rather is a separateoverlay.

FIGS. 23A and 23B illustrate a display formed by stitching or tiledimagery from six cameras pointed inward into the surgical field betweenthe surfaces of the retractor. In some embodiments, the cameras arepositioned on retractors to image a surgical site such that there islittle redundancy between sensors. The image processing module can beconfigured to stitch or tile imagery from the cameras to provide arelatively wide field of view of the surgical site. The differingpositions of the cameras on the retractors in FIGS. 23A and 23Bcorrespond to a retractor having an expanded size at a distal end andhaving the cameras positioned at a proximal end of the retractor (FIG.23A) and a retractor having a cylindrical shape with cameras positionedat a proximal end (FIG. 23B). These cameras can be used to create themain or background view shown in FIGS. 15 to 17. FIG. 20 discussed belowincludes an example retractor that may yield such a distribution offield of views.

Visualization Display with Movable Arm

FIG. 21L illustrates a front view of an embodiment of a visualizationdisplay with a display support platform attached to a movable arm. Thevisualization display 5000 includes a housing 5001 and a support pole5002 from which a movable arm 5003 extends. The movable arm 5003 canhave a C-shaped configuration with a distal end 5012 and a proximal end5013 as shown in FIG. 21M. The movable arm 5003 can have a display 5004mounted to the distal end 5012 of the movable arm. The display 5004 canbe mounted to the distal end 5012 of the movable arm 5003 through atilt/rotate device 5007. The device 5007 can allow the display 5004 tobe tipped, tilted, turned, and/or rotated to adjust the platform toachieve the required function and/or user preferences. In someembodiments the device 5007 can be a complex connection allowing fordetailed movement, articulation, and positioning of the display. Forexample, the device 5007 can include one or more hinges, arms, and/orjoints (e.g., ball and socket joints). In some embodiments, the movablearm 5003 at its proximal end 5013 can be attached to the support pole5002. In some embodiments, the support pole 5002 can be attached to theback surface 5008 of the housing 5001. FIG. 21Q illustrates aperspective view of an embodiment of a visualization display 5000 withan L-shaped support pole 5002. In some embodiments, the support pole5002 can be an L-shaped pole with a base 5009 and a stem 5010 as shownin FIG. 21Q. The base 5009 can attach to the back surface 5008 of thehousing 5001 and can extent outward at an angle approximatelyperpendicular to the back surface 5008 of the housing 5001. The stem5010 can extend vertically from the base 5009 at an angle approximatelyperpendicular to the base 5009.

In some embodiments, the movable arm 5003 can be attached to the supportpole 5002 through an attachment mechanism. The attachment mechanism caninclude, for example, a pivot attachment 5005 and ring 5006. In someembodiments, the ring 5006 can surround the support pole 5002,preferably surrounding the stem 5010 of the support pole 5002. The ring5006 can rotate about the support pole 5002, and can move verticallyalong the support pole 5002, preferably along the stem 5010. In someembodiments, a pivot attachment 5005 can be attached to the ring 5006extending outward from the ring 5006 at an angle approximatelyperpendicular to the support pole 5002. In some embodiments, the movablearm 5003 can be attached to the pivot attachments 5005. In someembodiments, the proximal end 5013 of the movable arm 5003 can beattached to the pivot attachment 5005 at an approximately perpendicularangle.

The ring 5006 can be moved vertically along the support pole 5002.Vertical movement of the ring 5006 can effectuate a vertical movement ofthe attached movable arm 5003 and the mounted display 5004. FIG. 21Lillustrates a front view of the visualization display 5000 showing themovable arm 5003 and the display 5004 are positioned at a verticalposition near the top of the stem 5010 of the support pole 5002. FIG.21N illustrates a front view of the visualization display 5000 similarto that described in FIG. 21L except the movable arm 5003 and thedisplay 5004 are positioned at a lower vertical position near the centerof the support pole 5002. The vertical movement up and down the supportpole 5002 of the movable arm 5003 and display 5004 can provide for thevertical position of the display 5004 to be adjusted to achieve thedesired performance or user preferences. The vertical movement of thedisplay 5004 can allow for accommodation of a surgeon in a seated orstanding position. Additionally, the vertical movement of the display5004 can accommodate the different heights of medical professionals. Insome embodiments, the ring 5006 can slide along the support pole 5002.The ring 5006 can have a locking mechanism (not shown) to fix the ring5006 to the support pole 5002 at the desired positioning. The lockingmechanism can include, for example, a screw, a knob, a clip, clamp, pin,or any combination of these as well as any other method that canfacilitate convenient locking or securing known in the art. In certainembodiments, friction between the ring 5006 and support pole 5002 can beadjusted. For example, the friction between the ring 5006 and supportpole 5002 can be adjusted through use of surface features (e.g.,roughness) and/or mechanical components (e.g., rubber pads or othermaterial additions to effect a change in friction between the supportpole 5002 and the ring 5006).

FIG. 21O illustrates a top view of an embodiment of a visualizationdisplay 5000 with the movable arm 5003 in a rotated position. Themovable arm 5003 as illustrated in FIG. 21M shows the display 5004 in afirst position directly in front of the housing 5001. FIG. 21Oillustrates a top view of the visualization display 5000 similar to thatdescribed in FIG. 21M, but FIG. 21O shows the movable arm 5003 and thedisplay 5004 rotated to a second position, outward from the housing 5001and the display 5004 is positioned to the side of the housing 5001. Themovable arm 5003 can rotate about the support pole 5002, preferablyrotating around the stem 5010 of the support pole 5002. In someembodiments, the movable arm 5003 can rotate up to about 90 degreesclockwise and counterclockwise from its unrotated position or firstposition where the display 5004 is positioned directly in front of thehousing 5001. In FIG. 21O, the movable arm 5003 is shown rotated about45° degrees clockwise from its unrotated position with the display 5004positioned directly in front of the housing 5001. In some embodiments,rotating the movable arm 5003 about the stem 5010 can provide the userwith more convenient access to the display 5004. In some embodiments,the visualization display 5000 can contain an internal counter balancesimilar to surgical microscopes to allow for the rotation of the movablearm 5003 without having to compensate for the additional weight.

In some embodiments, the movable arm 5003 can rotate about the pivotattachment 5005, as illustrated in FIG. 21P. The rotation about thepivot attachment 5005 can allow for the movable arm 5003 and the display5004 to rotate from one side of the housing 5001 to the opposite side.For example, FIG. 21L shows the movable arm 5003 positioned at a firstposition, the movable arm 5003 can then be pivoted about the pivotattachment 5005 and FIG. 21P shows the movable arm in a second positionwith the movable arm 5003 directly above the housing 5001. The movablearm 5003 can continue to pivot about the pivoting attachment 5005 untilreaching a third position as shown in FIG. 21Q on the opposite side ofthe housing from the first position. In some embodiments, the movablearm 5003 can be locked in a particular position by a locking mechanism.The locking mechanism can include, for example, a screw, a knob, a clip,clamp, pin, or any combination of these as well as any other method thatcan facilitate convenient locking or securing known in the art. In someembodiments, the movable arm 5003 can be locked in a particular positionby a self-locking mechanism in which the movable arm 5003 can bestabilized when placed at the appropriate position without the need fora locking mechanism. In some embodiments, the rotation of the movablearm 5003 about the pivoting attachment 5005 can facilitate rotation ofthe movable arm 5003 and display 5004 about the support pole 5002between one side of the housing 5001 (e.g., the left side with referenceto FIG. 21N) and another side of the housing 5001 (e.g., the right sidewith reference to FIG. 21N). Such rotation can permit left and righthanded users to adjust the position of the display 5004 and can enhanceversatility in set up of the display 5004. In some embodiments, thedisplay 5004 can maintain the display orientation during the rotation ofthe movable arm 5003 about the pivoting attachment 5005. For example, insome embodiments, the display orientation can be maintained by turningthe display 5004 about the device 5007 (e.g., see FIG. 21M). In someembodiments the display orientation can be maintained through imageprocessing. In addition, for use with the surgical visualization systemdisclosed herein, the display can be, e.g., a computer or instrumentdisplay used in other applications more generally. In theory, theapplications need not be even medical although anticipate medicalapplications such as disclosed.

Graphical User Interface

FIG. 24 illustrates an example graphical user interface that can be usedin embodiments of surgical visualization systems described herein. Thegraphical user interface can be used to receive user input and todisplay operational information to the user. In some embodiments, thegraphical user interface is displayed on a dedicated monitor systemseparate from the display system used to display images from the camerasassociated with the surgical visualization system. In some embodiments,the graphical user interface is incorporated into the same displaysystem that displays image data from the cameras associated with thesurgical visualization system, thereby providing a single display systemthat incorporates user interface elements and display output. In someimplementations, both are used, for example to give the surgeon and anurse or technician access to control system.

The GUI can be implemented on a monitor having a sterile touch screenthat is accessible to a surgeon, a surgical assistant, a scrubtechnician, or some other user or operator. The monitor with the GUI caninclude connectors that provide information to the GUI about theproperties and characteristics of the surgical visualization systemcoupled thereto. The GUI can receive tracking, position, and/ororientation information associated with cameras and surgical devices todisplay these in correct relative positions and orientations, forexample, on a schematic retractor graphic with a 3-D field of view conebeing displayed extending into the surgical site. In some embodiments, asmall, real-time video window from each camera can be displayed adjacentto a camera icon or included as the icon. The GUI can be used to selectone or more cameras to operate and/or to be displayed. The GUI can beused to position and/or orient each camera image on the display systemand to determine a size of the displayed image and/or a selected zoom ofthe image. In some embodiments, if a single camera is selected forviewing the default operation is to substantially fill the display withthe selected image. In some embodiments, if two or more cameras areselected for viewing the default operation is to position and orient thedisplayed images in correct geometric arrangement relative one toanother based on the relative locations of the camera or field of viewsetc. relative to a retractor frame coordinate system. In someembodiments, image data from cameras associated with a surgical tool canbe displayed using picture-in-picture techniques, as described hereinwith reference to FIG. 17, and overlaid on a background or main imageformed from one or more proximal wide field of view cameras. Thepicture-in-picture image can be a separate image or video stream fromthe other imagery. In some embodiments, the GUI can be used to selectwhether image data from the surgical tool will remain rotationallyconstant during use or whether it will rotate with tool rotation.

Accordingly, in various embodiments, the GUI can include cameraselection elements, positioned on the GUI where the camera selectionelements can provide a real-time video preview of the imagery acquiredby the camera. The GUI can provide a retractor graphic to displayrelative positions of cameras in the retractor coordinate system. TheGUI can provide control over LEDs using LED control element. The GUI caninclude an orientation icon to control whether the graphics are orientedrelative to gravity (as described herein), the patient, the display, thetable, etc. or to not orient the graphics (e.g., present imagery fromcameras without any relative and/or absolute rotation). The GUI caninclude an external source control so that imagery from external sourcescan be incorporated into the display. The GUI can include tool controlelements with functionality such as described in detail below. The GUIcan provide a method of controlling views on the display throughmanipulation of graphics on the monitor. For example, windows orthumbnails, presenting real time video from the cameras on the retractoror icons representing such real time video feed, can be moved about suchas moved more centrally, arrange with respect to each other, and changedin size, e.g., enlarged, as desired. The video windows or thumbnailsthemselves may be images from the camera or can be associated withimages from the camera. The video windows or thumbnails can be larger insome embodiments. A video camera feed can form the background of the GUIas well. The GUI can provide information to the user about a status ofvarious components such as cameras, LEDs, tools, orientation, and thelike. The GUI can be used to select whether displayed imagery, such asthe background or main image from a proximal camera, should change totrack a position of the surgical tool. In some embodiments, the GUIincorporates information received from a trackball interface that a usercan use to control view parameters. Also, the GUI can be split up intodifferent screens that can be selected by the user in some embodiments.

The graphical user interface can be used to provide tool and fluidicscontrol functions. In some embodiments, the GUI can be used to turn adrill on or off and to control a pressure associated with the drill.Similarly, the GUI can be used to operate a power Kerrison, a poweraneurysm clip applier, power forceps, bipolar & tissue forceps, and/orpower scissors controlling a state between on and off and controlling afluid or air pressure associated with the tool. In some embodiments, theGUI can be used to operate optics pressure washing, changing a statebetween on, off, standby, or auto. The GUI can be used to control afrequency of washing and a pressure. The GUI can be used to control anair-dry functionality, changing a state between on and off and changinga pressure with which it operates.

In various embodiments, virtual display technology is employed for theGUI. For example, images of the user's hand or a surgical tool may bedisplayed with the display device interacting with other images, e.g.,video windows or icons, displayed by the display device. Similarly, amotion detection system and/or gesture recognition system may beemployed to capture and identify movement of the user's hand or asurgical tool and display the movement and/or gestures on the displaydevice as well as associate those movements/gestures with inputinstruction. For example, using gesture control the surgeon can selectcameras or imagery for viewing, zoom in on a video stream, and/orposition video streams on the display.

In certain embodiments, for example, the viewing platform 9 shown inFIG. 21C with the cameras 18 mounted thereunder can employ one or moreof the cameras to act as gesture recognitions cameras 18 to image thesurgeon or user's hand located beneath the viewing platform. Thesurgeon/user can make gestures or otherwise move his or her hand, whichwill be imaged by the gesture recognition cameras. The gesturerecognition cameras can send signals corresponding to the images of handmovement to the display device that is visible to the surgeon/userpeering through the oculars. The image of the hand can be superimposedon images of graphics such as video windows, icons, buttons, etc. in thegraphic user interface. The user can thus move his or her hand so thatthe image of his or her hand, or a representation of the user's hand, asseen through the ocular, overlaps and/or interacts with (e.g., presses,moves, grabs, etc.) graphics on the GUI presented by the display deviceas seen through the oculars. Additionally, image recognition and/orgesture recognition processes can be applied to the image of the movinghand to analyze the location, movement, gesture and any combinationthereof to discern what instruction is being provide by theuser/surgeon. In some embodiments, other types of sensors may beincluded to assist in identifying the location of the hand and/or thegesture being made. A wide range of configurations are possible. In someembodiments stereo or 3D views may be provided by the one or morecameras. Additionally, although the hand has been discussed as providingthe recognizable gesture in this example, the system need not be solimited.

One advantage of a gesture recognition based GUI interface is that thesurgeon's hands may have blood or other bio-materials thereon. Thevirtual type GUI interface reduces the amount of direct contact of thesurgeon's contaminated hands with the visualization equipment andthereby reduces cleaning and/or sterilization requirements.

As discussed above, in some embodiments, unlike a conventional surgicalmicroscope, the viewing platform 9 is not a direct view device where thesurgeon or other user sees through the platform but is instead anindirect view device. The gesture recognition camera, however can alsobe used as a scene camera to view the patient and surgical site fromabove the patient, e.g., above the retractor by between 15-45 cm or morein some examples, as described herein with reference to FIGS. 21C-21K.The image of the patient and the surgical site can be displayed on thedisplay device and seen through the oculars. Even in embodiments wherethe gesture recognition system is not employed, one or more cameras 18beneath the viewing platform 9 can provide views of the patient andsurgical site to the display device for viewing through the ocular. Insome embodiments, stereo or 3D views may be provided by the one or morecameras.

FIG. 24B illustrates another example embodiment of a graphical userinterface for use with the surgical visualization systems disclosed anddescribed herein. The graphical user interface of FIG. 24B can be usedto display a relative orientation of cameras on a retractor, showing thefields of view as cones on the display. A real-time video feed of thevideo coming from the respective cameras can be displayed next to ornear the respective cameras or elsewhere on the GUI. In variousembodiments, real-time video feeds are shown in windows having areduced-size in comparison to the real-time video used by the surgeon toperform the tool manipulation and examine and interact with the surgicalsite. Accordingly, these reduced-sized windows may be sufficiently smallsuch that the surgeon may desire to increase their size at a later timeto glean sufficient detail from the video to assist in the surgicalprocedure. The user can manipulate the elements of the display to selectcameras for viewing, for adjusting video streams on the display,identifying or verifying alignment of cameras on the retractor, importimages or feeds for displaying, select and/or control surgical tools viaa toolbar, and the like.

In various embodiments such as shown, the GUI includes a model or CADimage of each camera on a graphic, model or CAD depiction of theretractor frame with a conical FOV for each camera, wherein the FOVdepiction can be based on tracking information, such as EM trackinginformation. In some embodiments, the user can verify a placement and/ororientation of the retractor cameras using this GUI as the GUI shows theorientation and FOV. In some embodiments, the user can manually orremotely adjust the placement and/or orientation of the retractorcameras using this GUI or other associated user interface tools. In someembodiments, the FOV displayed for the cameras can be semi-opaque or atleast partially transparent to allow a user to see other elements behindthe cameras to give a better sense of their relative location andorientation.

The GUI includes real-time video feeds corresponding to the video dataacquired with the various cameras on the retractor, the surgical tool,and/or auxiliary cameras. The real-time videos can be reduced-sizepresentations (e.g., reduced-size windows) of the acquired video whichcan be manipulated to select a video stream for viewing. For example, auser can select a video feed by selecting the appropriate reduced-sizewindow or presentation and placing it on the surgeon's binocular display(e.g., if the reduced size window is on the separate touchscreen inputand display device 13) and/or enlarging it on the surgeon's binoculardisplay 13 (e.g. if the reduced size window and CAD depiction of theretractor frame with a conical FOV's are on the surgeon's binoculardisplay).

The GUI includes a button or element that allows a user to importimaging data from other imaging modalities. For example, a user canimport CT, MR, C-arm, O-arm, and/or ultrasound images as well as imagesfrom anatomic databases. These images can be sized and positioned on thescreen, similar to the video feeds from the cameras of the surgicalvisualization system.

The GUI includes a positionable tool bar which enables control of opticswashing, optics drying, LED cooling, choice and control of tools, andthe like. The tools which can be controlled include, but are not limitedto, drill, Kerrison, bipolar forceps, scissors, aneurysm clip applier,etc.

The GUI can be configured to respond to multi-touch input, allowing theuser to virtually manipulate objects or video with their hands. The GUIcan also be configured to respond to other methods of input including,for example, voice recognition, gesture recognition, and the like.

The use of the GUI can be illustrated by the following example. Duringthe course of a procedure, a binocular surgical display can show one ormore real-time video feeds from one or more cameras in a size sufficientfor the surgeon to glean detail, which for the sake of this example isreferred to as a primary surgical view. See, for example FIGS. 15, 16,17, 20B, and 20D. If the user is interested in adjusting the viewsprovided, the user can perform a designated input, via touch input,gesture, voice command, or the like to bring up the GUI illustrated inFIG. 24B and select a different camera for presentation. The GUI canthen be presented on top of or instead of the primary surgical view(which for the sake of this example is referred to as a displayconfiguration view). The user then manipulates the elements of the GUIof the display configuration to configure the display (e.g., byselecting cameras to view, sizing the camera views, arranging/orientingthe camera views, etc.). Other functionality can be provided by the GUI,for example, the user can operate tools, import images, and the like.When finished, the display can revert to the primary surgical view whichreflects the changes made by the user on the display configuration view.In some embodiments, actions taken on the display configuration can beconfigured to immediately trigger actions or results on the primarysurgical view. For example, selecting a reduced size window showing areal-time video stream and resizing it about a threshold size can causethe display to show the primary surgical view with the selected cameraview full size or enlarged on the surgical view. In this way, forexample, the display can be configured to be immersive for the user dueat least in part to the ability of the user to perform a majority of thedesired functions without looking away from the display.

Transparent Rendering of Surgical Tools

In conjunction with the tracking information obtained as outlined above,the surgical tools can be rendered transparently or semi-transparentlyin the composite image. Electromagnetic tool trackers integrated with orfixed to surgical tools can provide position and orientation input tothe image processor. This information can be used to enable rendering oftool images as a transparent or semi-transparent overlay on thecontinuous composite display. The tools can, for example, be rendered aswireframe objects, or can be shown having a ghost-like transparency.This can provide the dual benefits of both indicating the position ofthe tool and maintaining an unobstructed wide view of the surgical site.In some embodiments, the dimensions and shape of the various tools canbe determined and catalogued prior to operation. The image processor maythen associate the observed tool with the predetermined size and shapestored in a database or library of tools in order to render a wire-frameor semitransparent image of the tool.

Camera Cleaning

Positioned within the body, the surface of the cameras or opticalelements, such as lenses, can become fogged or otherwise obstructed. Tomaintain visual clarity, the cameras can be cleansed while remaining inplace within the surgical site. One approach to cleansing cameras is toprovide pulses of fluid over the surface of the sensor or lens, therebyclearing any obstruction. Cleansing fluids may be, for example,distilled water, deionized water, or saline, among others. In someembodiments, these pulses may be brief, high-pressure, and low-volume.The pulse can be produced in a number of ways, for example by adiaphragm, actuated by a cam and motor in conjunction with a one-wayvalve. Fluid pressure can be supplied by air pressure in double spike IVbottle. A disposable diaphragm pump can be used to increase pulsepressure. In some embodiments, two pumps can be used to eliminateinterruption in pulse pressure. In some embodiments, passive hydraulicamplifiers can be used to increase the fluid pressure. In someembodiments, solenoids, piezoelectric actuators, or other techniques maybe used. A rolling edge diaphragm, Bourdon tube, or bellow can likewisebe used to produce the pulse. In one embodiment, a reed valve can beconfigured to alternate between air and saline operating at the naturalmechanical resonance frequency of the reed valve and associated fluidand air column dynamics. Pressure may then be maintained in the air andfluid circuits just below valve opening pressure, and an electricalsignal may increase pressure until reed valve opened, thus avoidingpulsating fluid tubing. In various embodiments, the fluid can be salineor other biocompatible liquid. In some embodiments, the lens elementsare configured such that a stop is affixed to a first lens elementwherein the stop covers a large fraction of the first lens element.Additionally, in some embodiments, the lens elements are configured suchthat a first element comprises a plano window. The stop can be locatedbehind the plano window or more lens elements and can be relativelysmall. In such embodiments, the light collected by the stop iscorrespondingly small such that the area of the plano window that shouldremain clean is relatively small. This configuration can facilitatecleaning the lens system. In some embodiments, the plano window issecured to the lens system using a structure that does not extend overthe top of the plano window so as to not interfere with mechanismsconfigured to clean the lens system. For example, the plano window canhave a step edge or be retained by a support member (e.g., a metal ringor an edge of the lens system housing) that extends along a side portionof the plano window without extending beyond the distal face of theplano window through which light is collected for the sensor.

In some embodiments, a high-pressure fluid pulse can be followed by ahigh-pressure pulse of air or gas in order to dry the surface of thecamera or lens and prevent salt deposits or image obscuration. Sourcesfor the high-pressure air may be, for example, hospital compressed airnitrogen systems, or compressed air or nitrogen tanks. The air pulse canbe actuated similar to the fluid pulse as described above. For example,a diaphragm actuated by a cam and motor in conjunction with a one wayvalve may be used. In some embodiments, a Venturi effect may be used togenerate the post-wash air flow. The Venturi effect depends on the sizeand shape of the port through which the fluid flows. In general, when afluid flows through a constricted section of pipe, the pressure isreduced. This low pressure can draw in additional outside air and cancause air flow following the fluid pulse. As discussed in more detailbelow, a proportional foot pedal can control actuation of the fluidand/or air pulses.

In some embodiments, the flex cable may include fluidic channels toconvey the air, gas, or liquid to the camera optics to provide forcleaning. Fluidic channels can also transport other fluids such aspharmaceuticals, saline for irrigation, fluorescent dyes, etc. to thesurgical site. Fluidic channels can also be provided for aspiration, toprovide egress of gases or liquids from the surgical site. The fluidicchannel containing flex cable may be an overlay or surrounding memberaffixed over the electronic flex cable, thereby allowing thefluid-carrying component to be disposable, where as the electronic flexcable with integrated optics module may be sterilizable and reusable.The distal end of the fluidic flex cable can contain an outer housingthat is secured over the imaging module. In some embodiments, it is theannular space and shape of the inner surface of said outer housing thatdirects the fluid and or fluid air pulse over the most distal surface ofthe optics for cleaning.

FIGS. 25A-C show an embodiment wherein an irrigation pathway 403 isprovided by an outer sheath 401 comprising a flex cable 405 includingfluidic channel containing flex cable and a portion that covers thesensor and imaging optics 409. The portion 402 of the sheath 401 thatcovers the sensor and imaging optics 409 can be shaped to provideconformal fitting yet leave a space 411 between the sheath 401 and thesensor and imaging optics 409 for air flow. A section 410 of the outersheath 401 forward of the imaging optics can be shaped to direct thefluid across the distal surface of the lens. In some embodiments, theouter sheath 401 that delivers the fluid can be a separable assemblythat can be added to or attached to the optical stack 409. In someembodiments, the fluid is delivered in the flex cable 405 that formspart of this sheath 401. This flex cable 405 is separate from the flexcable 407 that includes electrical connection for powering and receivingsignal from the camera. In some embodiments, when the separable assemblyis attached to the optical stack 409 and electric flex cable 407, thefluidic flex cable 405 assembly sits on top of the electrical flex cable407 and above the optical stack 409. In some embodiments, the outersheath 401 can be designed to snap onto the optical portion creating aseal around the optical stack 409 and then later detached. In variousembodiments, the detachable sheath 401 is disposable while the imagingoptics 409, sensor, and flex wire 407 connected thereto aresterilizable. In other embodiments, a fluid nozzle can be positioned onone side of the imaging optics, with an air nozzle on the other sidewith no sheath.

In some embodiments, the outer sheath 401 at the location of the cameraand imaging optics 409 includes an inner wall 412 in addition to anouter wall 414, both shaped as concentric right circular cylinders, oneinside the other. In some embodiments, the inner wall 412 which isclosest to the imaging optical 409 surrounds the optical stack so as toleave a gap 411 of air between the optical stack 409 and the cylindricalinner wall 412. This gap 411 advantageously facilitates air flow betweenthe optical elements (e.g., lenses) in the optical stack and therebyreduce the risk of condensation forming on the optical elements. Invarious embodiments, the outer sheath 401 is configured so as to allowfluid to enter the outer sheath 401 and be directed over the front mostsurface of the imaging optics 409. As described above, the deliveredfluid can be liquid, or gas, or a combination of both. In someembodiments, the irrigation pathway 403 can also be used to deliverother fluids such as pharmaceutical and fluorescent dies. In someembodiments, the irrigation pathway 403 can be used for aspiration andfluid egress.

Other configurations are possible. For example, in some embodiments, thefluid can be delivered by separate fluidic channels of the same flexcable that includes the electrical power and signal lines.

An additional approach to cleansing optical sensors and/or lenses is tomechanically ‘squeegee’ the surface. In one embodiment, as illustratedin FIG. 26, a fenestrated, rotary ring 501 with silicone rubber surfacethat can be arranged such that it is normally positioned with holes 503in front of the cameras 505. This approach may be particularly useful inembodiments in which the surgical device is a tubular retractor, forexample those used in minimally invasive spinal surgery. The cameras 505can be arranged along the annulus of the retractor. The rotary ring 501disposed over the cameras 505 may then be rotated such that siliconesweeps across the surface of the cameras 505 or lenses to clean them asa squeegee. The ring 501 may be rotated by a number of mechanisms. Inthe illustrated embodiment, a gear assembly 507 is disposed at theperiphery of the ring 501. In another embodiment, MEMS or pneumaticallyactuated individual blades may be arranged adjacent the front surface ofa camera module. The blade can be actuated to sweep across the frontsurface in a windshield-wiper fashion or otherwise, thereby cleansingthe camera.

Temperature Control

In various embodiments it may be advantageous to control the temperatureof certain components of the system. For example, heating the cameras toapproximately body temperature can reduce fogging effects duringsurgery. In some embodiments, the flexible cable and/or retractor bladescan include a heater to heat the cameras. However, excessive heat—suchas from LEDs or from an over-heated cameras—can damage surroundingtissue during use. Accordingly, in some embodiments a thermocouple maybe included in the device. For example, a thermocouple may be disposednear the cameras and/or LEDS. In some embodiments, a thermocouple sensorcan be used to provide temperature measurement and control of the heatedcamera or flexible cable. The thermocouple may be configured to providefeedback such that the system provides heating or cooling asappropriate.

Heating can be provided in a number of ways. For example, smallresistive heaters may be disposed within or near the cameras and/orLEDs. In some embodiments, a metallization disposed within the retractorcan provide the heating element. In some embodiments, the metallizationcan be a part of the flexible cable described above. Similarly, coolingcan be provided in a number of different ways. For example, the fluidpulses described above with respect to cleansing of the cameras may alsoprovide cooling effects. Accordingly, should a camera or LED becomedangerously hot, a brief, high-pressure pulse of fluid may be dispensedto decrease the temperature.

Foot Pedal with Feedback

A foot pedal may be used by the operator to actuate surgical toolsand/or to actuate cleansing of the cameras. For example, an operator maydepress a foot pedal to initiate the pulsing of liquid followed by thepulsing of air over the surface of the cameras. In some embodiments, theliquid pulse and the air pulse may be controlled by separate footpedals. With respect to surgical tools, a foot pedal may similarly beused for actuation. For example, a cutting tool such as scissors may bemoved from an open to a closed position by depressing the foot pedal. Aplurality of foot pedals may be provided, each associated with adifferent surgical tool. In some embodiments, a left foot pedal canprovide control of a surgical tool that would manually be controlled bythe operator's left hand, and a right foot pedal can provide control ofa surgical tool that would manually be controlled by the operator'sright hand.

In some embodiments, the foot pedal may actuate the deviceproportionally. For example, the degree to which the foot pedal isdepressed can control the amount of force applied to the closing ofscissors blades or forceps moving elements of an associated surgicaltool. In some embodiments, the foot pedal may provide sensory feedbackwith respect to the actuation of surgical tools. For example, theresistance to the closing of scissor blades can be measured andcommunicated to the operator by increased resistance to the depressionof the foot pedal. Such feedback can provide the operator with a senseof the resistance to cutting, despite the fact that the tool is actuatedindirectly via the foot pedal. The position of the foot pedal can bemeasured in a number of ways, for example by the use of an angularoptical encoder and/or a force-sensing resistor.

Switching Illumination On and Off

As noted above, in some embodiments one or more of the cameras may haveassociated illumination sources integrated nearby. For example, in someembodiments, each camera can include two adjacent LEDs, each on oppositesides, so that the LEDs illuminate the field of view of the camera. Thisillumination, however, can be problematic in instances in which one ormore of the cameras are oriented so as to at least partially faceanother camera. For example, if a first camera faces a second camera,adjacent LEDs on the second camera may effectively blind the firstcamera. The same principle applies to configurations in which theillumination sources are not associated with particular cameras. In anyarrangement in which a camera directly images an illumination source,there is a risk of blinding the camera, or at least degrading thequality of the image.

To address this shortcoming, the illumination sources and the camerascan be carefully timed or synched to avoid a camera directly imaging anactive illumination source. For example, an illumination source that isin the field of view of a camera can be controlled such that it is offor blocked while the camera is on, and conversely the illuminationsystem can be on while the camera is off or blocked. This arrangementcan be toggled rapidly, effectively strobing both the camera and theillumination source so as to reduce or eliminate the time during whichthe camera directly images an active illumination source. The camera andillumination source can be electronically controlled to allow forautomatic and rapid strobing. At a sufficiently high strobing frequency,the effect can be undetectable by an observer. The composite imagegenerated by stitching the views of the plurality of cameras can avoidany particular camera being blinded by the presence of an activeillumination source in its field of view.

As noted above, in some embodiments described herein there may be manycameras, and additionally there may be many illumination sources. Thestrobing principle noted above can be applied to such configurations ina similar fashion. For example, by appropriately timing each of theillumination sources and each of the cameras, images can be captured inwhich the active unblocked illumination sources are imaged less often,or in some embodiments not at all. Such timing can be realized in anumber of different approaches. For example, in one embodiment, eachcamera and its associated illumination source(s) can be turned on whileall other cameras and illumination sources are turned off. When onecamera and its illumination source(s) are then turned off, a next cameraand its illumination source(s) can then be turned on. This can continueat a rapid pace such that each camera is active for a brief time in agiven cycle. The cycle can be repeated continuously so that thecomposite image can effectively provide continuous wide field-of-viewvisualization.

In other embodiments, the strobing can be applied only to those cameraswhich directly image or have an illumination source in the field-of-viewof the camera. In some embodiments, optical recognition, positionaltracking, or other techniques can be used to determine when a camera ispositioned to image an illumination source. In some embodiments, oncethis condition has been determined, that camera may be modulated,synched, or strobed (e.g., switched on and off or blocked duringintervals of time and unblock during other intervals) in coordinationwith the associated illumination source, which also may be modulated orstrobed (e.g., switched on and off or blocked for certain intervals oftime and unblocked during other intervals). This approach can be appliedto configurations containing many cameras and many illumination sources.As noted above, the strobing may be affected continuously and rapidly,such that an observer is unable to detect the strobing, however, inother embodiments the rate of modulation or strobing need not be so fastas to be undetectable by the eye. In some embodiments, the rates ofmodulation, for example, may be below the absolute threshold of seeing,or below a difference threshold, where the measure or sensation of 2 ormore individual image modulations are not detected by the user.Moreover, brief flashes of the light sources and/or short activationperiods for the sensors need not be used. Yet, in such embodiments, theillumination may alternated in synchrony with the operation of imagesensors that are in the field-of-view of the light sources and viceversa.

In addition to direct blinding of a camera by an illumination source,specular reflections from instruments in the field of view of one ormore cameras may reduce image quality. For example, a particular sensormay receive an unwanted specular reflection from a particular tooloriginating at a particular illumination element. In variousembodiments, there are at least two possible responses: strobing andsynching of the offending illumination source and sensor, and/or anarea-of-interest calculation to reduce blooming or oversaturation in aportion of a sensors image, in particular a CMOS sensor. As the tools,sensors, and illumination sources can be known through tracking, invarious embodiments a global response to blooming may take the form oflook up table or global ray tracing algorithm to minimize over saturatedareas within an image or array of images as the tools are used withinthe illuminated scene.

White Balancing

As discussed elsewhere herein, a composite image can be formed bystitching together multiple, separate images from a plurality ofcameras. The images obtained from the separate cameras may, however,have different optical properties depending on a variety of factors. Forexample, the images obtained may vary in saturation, brightness,contrast, etc. depending on the camera and the configuration thereof andthe conditions under which the images are recorded. Stitching togetherimages of varying optical properties can create a patchwork effect inwhich the composite image can be degraded by the stark contrast inoptical properties from a region captured by one camera to the next.

In order to counterbalance such effects, the images recorded from theseparate cameras can be balanced or normalized. One such approachinvolves white balancing of the images from the cameras. For example, awhite or neutral colored object, such as a cylindrical fixture, can beinserted into the surgical space observable by some or all of thecameras. The color of this object can be used to normalize the opticalproperties (color gamut and gain) of the sensors and the differentimages, thereby allowing for a more seamless composite image generatedby stitching or tiling. In some embodiments, a multiple camera array cancolor balance composite images, wherein color balancing can include aglobal adjustment on one or more primary colors, red for example. Muchof the surgical scene is dominated by red and its corresponding channelmay be adjusted for continuity.

In some embodiments, the cameras could white balance on the mechanism,that holds the array of cameras inside a tubular retractor where onecamera image intersects another. Various other approaches can be used tonormalize the images obtained from the plurality of cameras. Theresulting composite image can therefore appear seamlessly stitched, suchthat the user is unable to readily determine the borders between oneimage and the next. Note that in some embodiments, the images may betiles such that discontinuities and/or gaps may exist in the compositeimage as a result of regions of the surgical site not imaged by thecameras.

In some embodiments, the item or mechanism used for color- orwhite-balancing can include a target to assist in aligning images fromvarious cameras. In some embodiments, the target can be a mechanism thatis used prior to inserting the retractor into the surgical site. Forexample, the mechanism can include a cylindrical or conical element witha substantially uniform color (e.g., white) and an asymmetric pattern.The retractor having a plurality of cameras can be positioned on themechanism such that the cameras image the mechanism. The imageprocessing module can calibrate the cameras based at least in part onthe imagery of the mechanism where calibration information can includewhite balancing and/or alignment information. The white balance can bebased at least in part on differences in color of the images of themechanism between the cameras. Alignment information can be based atleast in part on aligning the asymmetric pattern of adjacent images. Forexample, the white-balancing/alignment mechanism can be a uniformlywhite cylinder having asymmetric horizontal and/or vertical lines. Theimage processing system can use these features to align the imagesreceived from the various cameras by aligning the features within thecomposite image. Such a system can be used in association with asurgical tool as well. For example, the surgical tool can have a knowncolor and/or pattern and the image processing system can process imagesof the surgical tool to calculate a position and/or orientation of thesurgical tool as well as use the features of the surgical tool to alignimages from multiple cameras and/or color- or white-balance the images.

Example Embodiments of Imaging Modules

FIG. 27A shows some embodiments of an imaging module 700 comprisingwafer-scale optics (or wafer level optics) for use with a surgicaldevice. The imaging module 700 can include a stop 702 at a distal end ofthe module. In some embodiments, the stop 702 has a value at least about2.8 and/or less than or equal to about 3.3. The imaging module 700 caninclude one or more regions of material 704 a-d configured to act asoptical elements, such as lenses. The elements 704 a-d can be configuredto direct light and/or control distortion in the imaging module 700. Insome embodiments, the elements 704 a-d may comprise opticallytransparent material such as but not limited to plastic or glass (e.g.,molded glass). The elements may comprise, for example, monomers,polymers, or other compositions. In some embodiments, the elements 704a-d comprise resin. In some embodiments, the elements 704 a-d compriseacrylic, benzyl (meth)acrylate, (meth)acrylic acid copolymers ormulticomponent copolymers comprising benzyl (meth)acrylate,(meth)acrylic acid or other materials. The elements may also comprise awide range of other materials instead. As an example, and withoutlimitation, elements 704 a-d can have a refractive index that is atleast about 1.5 and/or less than or equal to about 1.7, or at leastabout 1.52 and/or less than or equal to about 1.65. As an example, someelements 704 a-d can have a relatively high dispersion and can have anAbbe number that is, without limitation, at least about 50, at leastabout 55, or at least about 60. Some elements 704 a-d can have arelatively low dispersion and can have an Abbe number that is, withoutlimitation, less than or equal to about 30, less than or equal to about25, or less than or equal to about 20. For some examples of wafer-scaleoptics (or wafer level optics), see U.S. Patent Pub. Nos. 2011/0063734to Sakaki and 2012/0134028 to Maruyama.

The imaging module 700 can include structural elements 706 a and 706 bconfigured to separate optical elements, provide mechanical durability,provide support to optical elements, maintain one or more gaps 712 inthe imaging module, or any combination of these. In some embodiments,the structural elements 706 a and 706 b can comprise a material that hasa relatively low index of refraction, that has a low coefficient ofthermal expansion, that is relatively impervious, or any combination ofthese properties. The structural elements 706 a and 706 b can beborosilicate glass, low-iron crown glass, or other similar material. Forexample, the structural elements 706 a and 706 b can be SCHOTT MEMpax®,BOROFLOAT® 33 borosilicate glass, D 263 T® eco borosilicate glass, AF32® eco aluminoborosilicate glass, or B 270® low-iron crown glasssupplied by Schott North America, Inc. or Duryea, Pa. Other materials,however, may also be used.

As an example, and without limitation, typical values for coefficientsof thermal expansion for structural elements 706 a and 706 b can bebetween about 2×10⁻⁶ K⁻¹ and 9.9×10⁻⁶ K⁻¹. As an example, and withoutlimitation, typical values for refractive indices for structuralelements 706 a and 706 b can be between about 1.4 and 1.6. As anexample, and without limitation, typical values for Abbe numbers forstructural elements 706 a and 706 b can be between about 50 and 70.Value outside these ranges, however, are possible.

The imaging module 700 can include an image sensor 708, as describedmore fully herein. The imaging module 700 can be electrically and/orphysically coupled to cable 710. In some embodiments, cable 710 is aflex cable, as described more fully herein. In some embodiments, themodule 700 has a field of view that is about 70 degrees.

In some embodiments, the imaging module 700 includes a moving elementconfigured to adjust a position and/or orientation of one or more lenselements. For example, the moving element can include a piezo, motor, orother transducer or actuator configured to translate one or more lenselements in one or more directions.

The imaging module 700 comprising wafer-scale optics can be manufacturedusing processes utilized in semiconductor wafer manufacturing. In someembodiments optically transparent material is deposited on a surface ofa substrate or structural support element (706 a, 706 b) such as acircular wafer like the wafers used for semiconductor device fabricationand is shaped to form a lens (704 a-704 d). The optically transparentmaterial may be shaped, for example, by imprinting, embossing, ormolding using a master. The master may be formed, in part by ion etchingin certain embodiments. In some embodiments, material is added to theoptically transparent material disposed on the substrate, material isremoved from the optically transparent material disposed on thesubstrate, or as mentioned above, the optically transparent materialdisposed on the substrate is shaped. In some embodiments, ion millingmay be employed. In some embodiments, multiple layers maybe be formedand joined during manufacturing. For example, multiple substrates orstructural support elements 706 a, 706 b may be provided, and lenses(704 a-704 d) formed thereon. The substrate or structural supportelements 706 a, 706 b can be stacked. In some embodiments, lenses 704 aare formed on a wafer which is cut to create separate support elements706 a having lenses disposed thereon. Similarly, lenses 704 b are formedon a different wafer which is cut to produce separate support elements706 b having lenses thereon. Support elements 706 a having lenses 704 athereon can be stacked on support elements 706 b also having lenses 704b thereon. Additional layers may be added as well.

Likewise, the wafer scale optics shown in FIG. 27A comprises a pluralityof substrates stacked on top of each other with spaces therebetween.Lenses having one planar surfaces adjacent to the planar surface of thesubstrate and one curved surface can, for example, refract and bendlight. As shown, such lenses having one planar surface formed on asubstrate and one curved surface can be formed on both sides of asubstrate (which as shown has two planar surfaces). Accordingly, in someembodiments, two curved surface are provide for a given substrate.

For example, as illustrated in FIG. 27B, a first layer 740 can be formedhaving a first thickness, d₁, and interlocking features 742. The firstlayer 740 may comprise the structural support element 706 a having layerof material formed thereon and patterned to produce the interlockingfeatures 742. A second layer 744 can be formed having a secondthickness, d₂, and interlocking features 746 that are complementary tothe first layer interlocking features 742. The second layer 744 maycomprise the structural support element 706 b having a layer of materialformed thereon and patterned to produce the interlocking features 746.The layer of material that is deposited may comprise monomer, polymer,glass or other material. The first and second layers 740, 744 can bestacked and interlocked to produce a portion of a wafer-scale opticsor/wafer level optics element. The first thickness, d₁, and the secondthickness, d₂, can be different. The height of the interlocking features742 and 746 can also be different. In some embodiments, multiple layers(e.g., two, three, four, five or more layers) are stacked to form theimaging module 700.

In some embodiments, wafers 750 such as illustrated in FIG. 27C arediced up to form the structural support elements 706 a, 706 b afterhaving the elements 704 a, 704 b formed thereon thereby producing theseparate multiple wafer-scale optics elements 752 that are stackable. Toalign the wafer-scale optics elements 752, fiducials features 754 can beincluded. In some embodiments, these fiducial features comprise theinterlocking features 742, 746 discussed above. Accordingly, fiducials754 on a proximal surface of a first wafer can be configured to matewith complementary fiducials 754 on a distal surface of a second wafersuch that the wafers interlock when aligned. In some embodiments,adhesives are included when stacking the layers. The adhesives can beUV-cured adhesives, thermal-cured adhesives, or other types ofadhesives. The plurality of the wafer-scale optics elements 752 can bealigned and locked together as described. In some embodiments, wafers750 such as illustrated in FIG. 27C are diced and then stacked.Alternatively, the wafers 750 can be stacked and then diced.Combinations of these approaches may also be used.

In certain embodiments, the fiducials 754 can provide stress relief tothe imaging module 700 thus formed. The structural support elements 706a, 706 b, having lenses 704 a, 704 b formed thereon may be subject tostress that causes the structural support element 706 a, 706 b to bow orotherwise misshapen. This stress may be caused at least in part, forexample, by differing coefficients of thermal expansion of variouscomponents of the wafer-scale optics elements 752 such as of thestructural support element 704 and the lens 706. In various embodiments,the fiducials 754 can be used to apply forces on proximal and distalsurfaces of the wafers and structural support element 706 a, 706 b thusassisting the structural support element 704 and the lens 706 thereon tosubstantially to retain its shape. In some embodiments, the fiducials754 comprise monomers, polymers, glass, or other material deposited onproximal and/or distal surfaces of a wafer 750. The fiducials 754 can beconfigured to reduce or eliminate stress-induced warping due at least inpart to differing coefficients of thermal expansion of various elementson the wafer and/or in the imaging module 700. In various embodiments,the stress contributed by the fiducial layers 754 on opposite sides ofthe structural support element 704 largely offset each other to reducethe amount of bowing. In some embodiments, the amount of materialforming the fiducial 754 may exceed the amount of material forming thelens such that the stress induced by the fiducial on both proximal anddistal sides of the structural support element exceeds and overwhelmsthe stress induced by the lens. In some embodiments, the stress inducedby the lens 704 and the fiducials on the same side of the supportelement 706 is substantially similar to the stress induced by thefiducial layer on the opposite side of the support element 706.

In certain embodiments, the structural elements 706 a, 706 b (referringto FIG. 27A) in the imaging module 700 comprise substrates having one ormore holes 714 extending from a proximal surface to a distal surface ofthe substrate. The holes 714 in the substrate can allow fluids tocommunicate between regions or gaps 712 between the structural elements706 a and 706 b. The holes 714 can be formed by drilling through thesubstrate or using a chemical machining process during manufacture ormay be formed using other techniques. With reference to FIG. 27C, theholes 714 can be cut in the plurality of wafers 750. As described above,the wafers 750 can be cut and the substrates 706 formed therefrom can bestacked and interlocked, creating compartments 712 between thesubstrates 706 that otherwise would provide a barrier preventing fluidiccommunication between compartments. The holes 714, however, thus canprovide for fluidic communication between these compartments. Fluidiccommunication can reduce or eliminate condensation on optical elementsin the imaging module 700, e.g., on the lenses 704 during use orprocedures where elements are exposed to fluids and/or changes intemperature that facilitate or promote the formation of condensation.

FIG. 28 shows some embodiments of an optical prescription for an imagingmodule 800 having a field of view less than or equal to about 70degrees. The imaging module 800 can include a stop 801 on a distalportion of the imaging module in combination with four or more lenses.In some embodiments, the imaging module 800 includes the stop 801, apositive lens, a negative lens, and a plurality of positive lenses. Theimaging module 800 can be relatively compact while providing a moderatefield of view and resulting in relatively low distortion (e.g., lessthan about 10% distortion). In some embodiments, the first element 802is a positive lens having a relatively high index of refraction andrelatively moderate dispersion. The first element 802 can be combinedwith a lens having a relatively high index of refraction and relativelyhigh dispersion. Lens elements 806 and 808 can be positive lenses andconfigured to shorten an optical path and correct for distortionsintroduced by previous lens elements. The imaging module 800 cancomprise wafer-scale optics. The imaging module 800 can be used with asurgical device.

As shown in FIG. 28, the imaging module 800 comprises lens elements 802,804, 806, 808, 810, and can be configured to form an image at 812. Therespective lens elements can have properties, such as distal andproximal radii, thickness, distance to the next element, index ofrefraction (n), Abbe number (V), and the like. Table 1 lists examplevalues for various properties of the optical elements 802 through 810.For reference, a distal surface of a lens element is the surfacefurthest from the image 812, and a proximal surface of the lens elementsis the surface closest to the image 812. Furthermore, a thickness is thedistance from the distal surface to the proximal surface in a particularlens element. Finally, the column labeled distance in the table refersto the distance from the proximal surface of the element in that row tothe next element (i.e., the element in the next row). In the exampleshown in FIG. 28 and Table 1, the stop is at the first surface and hasan aperture of 0.125. The values in the table are normalized to producean effective focal length of 1. Note that in this design, the aperturestop is in front and the lens, in order, from front to back arepositive, negative, positive, and positive. This design may be suitablefor providing fields-of-view less than 70 degrees, e.g., between 50-70degrees.

TABLE 1 Lens Distal Proximal Thick- Dis- Element R R ness tance n VElement 802 0.823 −3.216 0.235 0.089 1.79 47.5 Element 804 −1.04 1.040.07 0.13 1.76 26.5 Element 806 −2.94 −0.8 0.255 0.012 1.79 50.0 Element808 1.3 6.055 0.44 0.341 1.79 50.0 Element 810 infinity n/a 0.25 0.041.52 58.6

In certain embodiments, the imaging module 700 includes a lens groupcomprising two or more lenses (806, 808) at a proximal end thatconcentrate or focus light, similar to a condenser lens group. In someembodiments, this group comprises two positive power lenses. In someembodiments, these lenses may comprise positive power lens 806 having aconcave distal surface and a convex proximal surface and a positivepower lens 808 having a convex distal surface and a concave proximalsurface as shown. Accordingly, the vertices of these lenses 806, 808 mayface each other and be in close proximity or even be in contact witheach other. Although the stop is shown forward the first lens 802, incertain embodiments, the stop 702 can be located between lens elements.

FIG. 29A shows some embodiments of a wide field-of-view imaging module900 with a buried stop 904 for use with a surgical device.Advantageously, some embodiments include the buried stop 904 allowingfor a relatively wider field of view. In some embodiments, aspheresand/or more elements are included in the imaging module 900 to correctfor distortion. In various embodiments, the imaging module 900 mayinclude imaging optics comprising a negative distal lens group 902having one or more lenses that produce a total optical power for thatgroup that is negative and a positive proximal lens group 906 having oneor more lenses that produce a total optical power for the group that ispositive. The negative distal lens group 902 can be relatively stronglycurved. A plano-concave lens can be used to facilitate fabrication,alignment, and/or cleaning, among other advantageous features. In someembodiments, the negative distal lens group 902 can include aplano-concave lens that has a relatively large negative optical power, arelatively high index of refraction, and a relatively moderatedispersion. In certain embodiments, the negative distal lens group 902has a relatively high index of refraction and a relatively highdispersion. Some examples of lens elements suitable for use in thenegative distal lens group 902 are SF10, SF14, SF57, supplied by SchottNorth America, Inc. of Duryea, Pa., and LAH58, LAF-21, LAFN31, LASF41,and LASF44 supplied by Ohara of Rancho Santa Margarita, Calif. In someembodiments, the negative distal lens group 904 includes lens elementsconfigured to correct for distortion introduced by lens elements havinga relatively strong curvatures or a relatively high negative opticalpower. In certain embodiments, the positive proximal lens group 906 isconfigured to correct for distortions introduced by the negative distallens group 904. The imaging module 900 can include a stop 904 whereinthe negative distal lens group 902 is on a distal side of the stop 904and the positive proximal lens group 906 is on a proximal side of thestop 904. In some embodiments, the stop 904 can have an f-stop valuethat is, for example, at least about 3 and less than or equal to about10 and may be at least about 4 and less than or equal to about 8. Insome embodiments, the imaging module 900 can include one or more afocallens groups but may be non-afocal as well. The imaging module 900 caninclude a camera 908, as described more fully herein. The imaging module900 can be configured to have a wide field-of-view. For example, theimaging module 900 can be configured to have a field-of-view that is atleast about 90 degrees and/or less than or equal to about 120 degrees.

In some embodiments, the wide field-of-view imaging module 900 comprisesnon-wafer-scale optics combined with wafer-scale optics on a distal sideof the stop 904. The wide-field of view imaging module 900 can includeadditional optics on a proximal side of the stop 904. In certainembodiments, the negative distal lens group 902 comprisesnon-wafer-scale optics and the positive proximal lens group 906comprises wafer-scale optics. In some embodiments, the negative distallens group 902 comprises wafer-scale optics having a stack of negativelens elements on a distal side of the stop 904. The wide field-of-viewimaging module 900 can include optics on a proximal side of the stop 904comprising wafer-scale optics or non-wafer-scale optics.

FIG. 29B shows an example embodiment of an optical assembly 950comprising an afocal module 955 coupled to an optical imaging module 960for use with a surgical device. The afocal module 955 can be added to anexisting optical imaging module 960 to create a combined opticalassembly 950 configured to achieve desired optical properties. Thepreviously existing optical imaging module 960 may comprise, forexample, a wide field-of-view, imaging module that otherwise would forman image on the sensor 975. In some embodiments, the optical assembly950 has a different field of view than the optical imaging module 960due at least in part to the afocal module 955. For example, whencompared to the optical module 960, the combination of the afocal module955 and the optical module 960 can have a broader or narrower field ofview as determined, at least in part, to the optical properties of theafocal assembly 955. In some embodiments, the optical assembly 950includes an optical filter. The optical filter can be included betweenthe afocal module 955 and the optical module 960, the optical filter canbe part of the afocal module 955, or the optical filter can be a part ofthe optical module 960. The optical assembly 950 has a stop 970 locatedin the optical module 960. In some embodiments, the stop 970 can belocated in the afocal assembly 955 or between the afocal assembly 955and the optical module 960.

The afocal assembly 955 includes optical elements 957 a, 957 b. A firstoptical element 957 a can have a negative optical power. A secondoptical element 957 b can include optical elements that, when combinedwith the first optical element 957 a, form an afocal module 955. Theoptical elements 957 a, 957 b can be wafer-scale optics, non-wafer-scaleoptics, or a combination of these. The second optical element 957 b ofthe afocal assembly 955 includes a lens element 965 having aconfiguration wherein a central circular portion 966 has a negativeoptical power and an annular peripheral portion 968 has a positiveoptical power. The lens element 965 can be configured to alter adirection of propagation of peripheral rays more than of central rays.The lens element 965 can be used to correct for distortions introducedby the first optical element 957 a. The lens element 965 can bepositioned to increase or maximize corrections to distortions. Forexample, the lens element 965 can be positioned near the image plane 975or near the first optical element 957 a. The lens element 965 can bepositioned before the stop 970.

As shown, the optical element 957 b includes lens on the proximal anddistal sides thereof. The lens shown on the distal side may comprise apositive power lens in some embodiments. Other configurations, however,are possible.

The optical elements 957 a, 957 b can be aspherical, spherical, or haveanother shape. The optical module 960 can include one or more opticalelements 962 wherein the optical elements comprise wafer-scale optics,non-wafer-scale optics elements such as glass, or a combination ofthese. In some embodiments, the afocal module 955 comprisesnon-wafer-scale optics optical elements made of glass. In someembodiments, the optical module 960 comprises wafer-scale optics. Theoptical assembly 950 can thus be configured as a combination ofnon-wafer-scale optics optical elements and wafer-scale optics. Theafocal module 955 can include spherical lens elements. The lens elementsof the afocal module 955 can comprise materials having a relatively highindex of refraction (e.g., without limiting the indices of refraction,typical values can be between about 1.55 and about 1.7) and relativelyhigh dispersion (e.g., without limiting the Abbe number, typical valuescan be at least about 50, 55, or 60) or materials having a relativelylow index of refraction (e.g., without limiting the indices ofrefraction, typical values can be between about 1.3 and about 1.6) andrelatively low dispersion (e.g., without limiting the Abbe number,typical values can be less than or equal to about 30, 25, or 20). Insome embodiments, the optical assembly 950 can have a relatively lowf-number compared to other wafer-scale optical assemblies wherein thef-number is set to provide a relatively greater depth of field comparedto wafer-scale optics suitable for use where the subject is relativelyfar from the image sensor. In some embodiments, adding the afocal module955 to the optical module 960 reduces the focal length, therebyincreasing the field of view. Although module 955 has been described asan afocal module 955, this module need not be afocal.

As described above, the module 955 may advantageously be added to apre-existing optical imaging module 960. In this manner, thefield-of-view may be altered. For example, the pre-existing opticalimaging module 960 may have a modest field-of-view, e.g. of between50°-70°. However, addition of the add-on module 955 may increase thefield-of-view to beyond 70° such as to 90°-120°.

FIG. 29C shows an imaging module 980 comprising optical elements 982 aand 982 b configured to change properties of the imaging module 980. Forexample, a first interchangeable optical element 982 a can comprise aplano-plano block having an optical path length of d. This blockprovides for a substantially straight linear optical path as shown. Asecond interchangeable optical element 982 b can comprise a block havingfeatures 983 that change the direction of the optical path within theinterchangeable optical element 982 b. These features 983 may comprise,for example, refractive surfaces that redirect light towards a side ofthe optical element 982 b. These refractive index surfaces compriseinterfaces between different section (three shown) of the prism 982 boptical elements. Although three are shown, more or less may beincluded. The second interchangeable optical element 982 b can have anoptical path length of d, same as the first interchangeable opticalelement 982 a. By switching between interchangeable optical elements 982a and 982 b (e.g., at the manufacturing stage), a viewing angle of theimaging module 980 can be changed. Having similar optical path lengthsreduces the design implications of the change. The imaging module 980can include a first negative lens or lens group 984 that can be moved orreoriented when changing between interchangeable elements 982 a, 982 b.In some embodiments, the negative lens or lens group 984 can be a partof the interchangeable optical element 982 a, 982 b. The imaging module980 includes a proximal positive lens group 986. The lens group 986 canhave a total power that is positive or negative and may comprisewafer-scale optics.

FIG. 29D shows an example imaging module with optics providing a viewingangle relative to a surgical tool axis. The example imaging module canbe used with a stereo optical sensor to provide a narrow profile along asurgical device (e.g., a retractor blade) while providing a viewingangle, θ, that is not parallel to the axis of the surgical device norparallel to the surgical device. This can allow for the use of arelatively large image sensor with little or minimal obstruction of asurgical site due to an imaging module protruding from a surgical devicesuch as a retractor. The imaging module can include a sensor, whereinthe sensor can be configured for monocular imaging or the sensor can bea divided sensor configured to provide stereo imagery. The imagingmodule can include a prism configured to direct light to the sensorwhere the incoming optical axis is substantially parallel to an axis ofthe surgical device. The imaging module can include optics configured tofocus light from the surgical site onto the sensor. The imaging modulecan include redirection optics configured to direct light from a regionof interest to the prism where the light comes generally from an angle,θ, relative to the axis of the surgical device. Accordingly, the imagingmodule can provide a viewing angle, θ, that is at least about 30degrees, at least about 45 degrees, at least about 70 degrees relativeto the axis of the surgical device. In some embodiments, an auto focusmechanism can be added between the sensor and the prism which can allowa user to control a focus of the imaging module and/or a fasterf-number.

In various embodiments other types of prism elements may be used insteadof the prism element 982 b shown in FIG. 29C. For example, the prismelement may be formed by cementing together smaller prism sections tocreate a “compound” or “compact” prism or alternatively may the prismelement may comprise a deviating prism that is a single homogenous pieceof transparent material having outer surface shaped to obtain thedesired beam direction. This material may be glass in variousembodiments. An optical element that redirect light using multiple andin some embodiments an even number of reflections, e.g., 2, reflectionscan offer benefits with respect to parity or inverting and/orre-orienting an image upright.

Such interchangeable optical elements 982 a, 982 b can be used forcameras on surgical tools to change the view of the camera from directlyin front of the camera, for example, directly forward the distal end ofthe tool. Instead, a view angle towards the side of the tool can beprovided by interchanging the optical element 982 a, 982 b.

FIG. 30A shows some embodiments of an imaging stack 1000 comprisingnon-wafer-scale optics 1002 in combination with wafer-scale optics 1004and 1006 for use with a surgical device. The imaging stack 1000 includesa cover glass 1001. The cover glass 1001 can be made of a materialhaving desirable optical and/or mechanical properties, such as sapphire.The cover glass 1001 can be a plano-plano optical element having a stop1012 bonded thereto or formed thereon. The combination of the coverglass 1001 and stop 1012 can provide for a relatively small area of thecover glass 1001 to be exposed, thereby facilitating cleaning. In someembodiments, the non-wafer-scale optics 1002 include one or morecomponents made of glass, polycarbonate, sapphire, or other materialhaving desired optical properties. In some embodiments, thenon-wafer-scale optics 1002 have one or more lenses that produce a totaloptical power that is negative. For example, the non-wafer-scale optics1002 can include a lens with a negative optical power having arelatively strong curvature typically not produced for wafer-scaleoptics. The imaging stack 1000 can include a first wafer-scale opticalelement 1004. The first wafer-scale optical element 1004 can include oneor more components 1008 configured to provide optical correction,provide a desired focal length, provide an afocal system, or the like.In some embodiments, the one or more components 1008 are configured tocompensate for distortion in a wide field-of-view system. The firstwafer-scale optical element 1004 can include registration marks,structures, fiducials, or features 1010 configured to aid in subsequentimage processing or mounting/registering non-wafer-scale optics 1002with minimal or reduced active optical alignment and increased alignmentprecision. In some embodiments, the combination of non-wafer-scaleoptics 1002 and first wafer-scale optical element 1004 provides a totaloptical power that is negative. The imaging stack 1000 can include apositive proximal wafer-scale optical group 1006 having one or moreoptical elements that produce a total optical power for the group thatis positive.

FIG. 30B shows some embodiments of an imaging stack 1000 comprisingnon-wafer-scale optics 1002 in combination with wafer-scale optics 1004and 1006 for use with a surgical device. In some embodiments, theimaging stack 1000 includes a stop 1012 positioned between the firstwafer-scale optical element 1004 and the positive proximal wafer-scaleoptical group 1006. The positive proximal wafer-scale optical group 1006can include an optical element 1009 wherein a radial central portion1011 of the optical element 1009 has a negative optical power and aradial peripheral portion 1013 of the optical element 1009 has apositive optical power. In some embodiments, the optical element 1009can be configured to alter a direction of propagation of peripheral raysmore than of central rays. The optical element 1009 can be used tocorrect for distortions introduced by optical elements in thenon-wafer-scale optics 1002, the first wafer-scale optics, and/or thepositive proximal wafer-scale optical group 1006. The optical element1009 can be positioned to increase or maximize corrections todistortion. The imaging stack 1000 includes an image plane 1010.

In some embodiments, wafer scale optics (or wafer level optics) mayinclude an aperture therein provided, for example, by the manufacturer.Nevertheless, in various embodiments, another aperture that can be usedas the stop can be placed between a non-wafer scale optical element andthe wafer scale optics having the aperture therein. For example, a stopmay be included between a negative power distal lens group comprisingone or more non-wafer scale optical element and a proximal lens groupcomprising wafer scale optics.

In some embodiments, the cameras and/or optical systems described hereincan include MEMS configured to change optical properties of the opticalsystem, such as, for example, a focal length, a magnification, apointing angle, a field of view, or any combination of these. In someembodiments, an actuator can be used to drive a low power opticalelement substantially along the optical axis. The optical element can bedriven at a rate such that the full sinusoidal cycle of focus iscompleted at a greater rate than the human critical flicker fusion rate(e.g., about 60 Hz). This can increase a depth of field of the camera oroptical system. The actuator can include, for example, a voice coil, amoving magnet, a piezo, or other such actuators. In some embodiments,increasing the depth of field in this manner can be accomplished usingCOTS MEMS auto-focus modules driven by a sine wave having a frequencythat is greater than or equal to about 60 Hz. In some embodiments, theauto-focus actuator can be driven such that the focal plane or median ofincreased depth of focus volume tracks a desired portion of a surgicaltool, such as a tip of the surgical tool, where a position of thesurgical tool is acquired using any of the tracking methods describedherein, such as EM tracking.

In some embodiments, cameras can be positioned at different depthswithin the surgical site along a retractor. The image processing systemcan use the varying depths of field from the cameras at different depthsto increase an output depth of field to provide focused imagery of agreater portion of the surgical site. For example, a retractor can havetwo or more rings of cameras and the cameras at each level can provideimage data to the image processing system. The image processing systemcan combine overlapping image data (e.g., by tiling and/or stitchingimages) to provide an output image having a greater depth of focus. Anexample of a system with cameras at two depths is illustrated in FIG.21.

In some embodiments, surgical devices, such as surgical tools orretractors, can include right-angle prisms to direct light from a sceneonto an optical sensor. This can be used so that the optical sensor andassociated optics can be housed within the surgical device or otherprotective housing thereby providing increase working room for thesurgeon that is not obstructed by cameras and to reducing possibledamage to cameras from use.

In some embodiments, LEDs associated with cameras can produce a lightcone that is substantially similar to the field of view of theassociated camera. Such a light beam may be used in set-up to positionand/or orient the cameras. This process may be employed when the camerasare in the initialization platform.

FIGS. 31A and 31B respectively show top and side views of someembodiments of an imaging module 1100 comprising an imaging stack 1106,sensor layer 1114, via layer 1104, and flex layer 1110. The via layer1104 can include vias configured to permit liquid or electrical cablesto pass through. The via layer 1104 can be coupled to the flex layer1110, which can comprise a flex cable according to some embodimentsdescribed herein. The flex layer 1110 can include a plurality ofchannels 1112 configured to transmit liquid or house electrical cable.The imaging module 1100 can include an image stack 1106, such as imagingstacks described herein with specific reference to FIGS. 27A-30B. Theimaging module 1100 can include a sensor layer 1114 configured to housean image sensor, as described more fully herein. The vias 1102 on thevia layer 1104 can be configured to transmit liquid to or from the flexlayer 1110. In some embodiments, the vias 1102 are coupled to orcombined with cleaning systems as described herein. The vias 1108 can beconfigured to pass electrical cables from the flex layer 1110 to thesensor layer 1114. In some embodiments, the flex layer 1110 comprisesmore than one flex cable with at least one flex cable configured tohouse electrical cables and at least one flex cable configured totransmit fluid to and from the imaging module 1100.

As noted above, in some embodiments, illumination sources such as LEDsmay be positioned near the optical sensors. For example, in the case ofa retractor, in some embodiments optical sensors positioned on theretractor blades may each have one or more corresponding LEDs positionedadjacently. However, in other embodiments, the LEDs need not bepositioned adjacent the optical sensors. For example, the LED or otherillumination source may be positioned at some distance from the opticalsensors, but light may be directed from the LED to an area near theoptical sensor using a one or more light guides or mixers.

FIGS. 32A and 32B illustrate two embodiments of light guides/mixers foruse with imaging modules. As shown in FIG. 32A, a light guide 1201 hasan illumination source 1203 disposed adjacent the first end 1205 of thelight guide 1201. In various embodiments, the light guide 1201 can bemade of resin, glass, plastic, or any other transparent materialsuitable for propagation of light therein. The thickness of the lightguide 1201 can decrease from the first end 1205 to the second end 1207,creating a tapered profile. In various embodiments, light entering thelight guide 1201 at the first end 1205 propagates through the body ofthe light guide 1201 by total internal reflection. In implementations inwhich the light guide 1201 is tapered, light guided in the light guide1201 will propagate by total internal reflection until it is ejected bythe light guide 1201 at an oblique angle relative the light guide 1201.In some implementations, the rearward surface 1211 of the light guide1201 can have a reflective surface so as to reflect light emitted fromthe light guide plate 1201 such that the light is turned and output fromthrough light guide 1201 and emitted from the forward surface 1213. Invarious embodiments, the reflective surface can be metallic. In someembodiments, a film having a lower index of refraction than that of thelight guide 1201 may be positioned over the rearward surface 1211 tosupport the propagation of light therein.

As shown in FIG. 32A, light can be emitted from the forward surface 1213of the light guide 1201 along the length of the light guide 1201. Invarious embodiments, the light emitted from the forward surface 1213 canbe substantially uniform across the entire length of the light guide1201. In some embodiments, the light emitted from the forward surface1213 can be emitted across substantially the entire forward surface1213. In some embodiments, the emitted light may exhibit directionality.In other embodiments, the emitted light may be dispersed substantiallyuniformly.

FIG. 32B shows a light guide 1201 and illumination source 1203 similarto that of FIG. 32A. However, as shown in FIG. 32B, the light guide 1201has a substantially rectangular profile along part of its length, with atapered profile near the second end 1207. Additionally, a reflectivelayer 1215 is disposed over a portion of the forward surface 1213 of thelight guide 1201. The reflective layer 1215 can be, for example, ametallic layer. In other embodiments, the reflective layer 1215 can bereplaced with a dielectric film that is substantially non-transmissive.In various embodiments, a reflective opaque layer is used for the layer1215. As a result of the reflective layer 1215, light propagating withinthe light guide 1201 that would otherwise be emitted from the forwardsurface 1213 is reflected by the reflective layer 1215 and continues topropagate within the light guide 1201. As illustrated, the reflectivelayer 1215 does not extend along the full length of the light guide1201. The portion of the forward surface 1213 nearest the second end1207 of the light guide 1201 is not covered by a reflective layer 1215.As a result, light propagating within the light guide plate 1201 can beemitted from the forward surface 1213 in the region not covered by thereflective layer 1215. In other embodiments, the light guide 1201 may betapered along part of its length such that the thickness increases fromthe first end 1205 towards the second end 1207, until it the portionnearest 1207, at which point the thickness decreases as with theillustrated embodiment.

In contrast to FIG. 32A, the light guide of FIG. 32B emits light fromthe forward surface 1213 only in one region of the light guide 1201.Accordingly, the emitted light can be directed to a specific region,resulting in a spotlight effect depending on the directionality of theemitted light. As noted above, the configuration illustrated in FIG. 32Acan provide substantially uniform emission across the entire forwardsurface 1213 of the light guide, resulting in a wider floodlight effect.

In either of the illustrated embodiments, the illumination source 1203may include one or more light emitters such as light emitting diodes. Insome embodiments, the illumination source 1203 can include multiple LEDsof different colors. For example, in various embodiments theillumination source 1203 can include red, green, and blue LEDs. Lightfrom each of the different colored LEDs propagates through the lightguide 1201 and in the process the light is mixed. As a result, lightemitted from the forward surface 1213 can be mixed, for example suchthat the red, green, and blue light produces an emitted white light fromthe forward surface 1213 of the light guide 1201. In variousembodiments, additional colors, such amber, near infrared, or othercolors can be added. The inclusion of amber light may provide additionalcolor between green and red where the LED spectrum from phosphor whitesor discreet RGB modules may have a gap.

In some embodiments, an optical film may be positioned over the forwardsurface 1213 of the light guide 1201. For example, a diffusing film canbe positioned to aid in color mixing and/or to provide a more uniformemission profile. In some embodiments, the optical film can have otherproperties, for example the optical film can operate to focus thedirection of the emitted light over a narrower range. In variousembodiments, an optical film comprising a color filter can be positionedover the forward surface of the light guide. As with the reflectivelayer 1215 illustrated in FIG. 32B, in various embodiments anyadditional optical film may not necessarily extend across the entireforward surface of the light guide. In various embodiments, the filmcould have a diffractive or other pattern for improving or modifying theoutput distribution.

Although the illustrated embodiments show a tapered light guide, invarious embodiments the light guide may assume other profiles. Forexample, in some embodiments the light guide can be substantiallyrectangular in profile, and may include light-turning features on one ormore surfaces to redirect light propagating within the light guide suchthat the light is emitted out the forward surface. Such light-turningfeatures may be prismatic (e.g. facets) or diffractive (e.g., aholographic film), and may be formed integrally with the light guide ormay be included in a separate optical film placed over one or moresurfaces of the light guide. In various embodiments, the light guide1201 can include prismatic light-turning features to redirect light outof the light guide 1201, rather than relying solely on the taperedprofile.

FIGS. 33A and 33B illustrate top and side cross-section views,respectively, of a retractor blade with a light guide and illuminationsource integrated therein. As shown, the distal portion of one retractorblade 1217 can have positioned thereon a light guide 1201 with anillumination source 1203 positioned adjacently. In the illustratedembodiment, the illumination source 1203 comprises three separate LEDs.As noted above, in various embodiments, different color LEDs such asred, green, and blue LEDs may be used to inject light into the lightguide 1201, where it is then mixed and ejected as white light. In someembodiments, red, green, and blue LEDs may be used in conjunction withan amber LED. Such combination may provide a desirable look for redtissue that is viewed by the cameras. In some embodiments, white LEDsmay be used. Light sources other than LEDs may also be used. Asdescribed above, light from the illumination source 1203 propagates inthe light guide 1201 until emitted from the forward surface of the lightguide 1201.

The retractor blade 1217 may be used in conjunction with multiple otherretractor blades, each similarly equipped with an illumination source1203 coupled to a light guide 1201. Each retractor blade may include oneor more optical sensors (not shown). Each light guide 1201 on aretractor blade 1217 can provide illumination for a certain area,depending on the orientation of the retractor blade and the propertiesof the light guide and illumination source. The light guide 1201 may beimbedded within or incorporated as part of the retractor blade or may beremovably attached thereto. For example, the light guide may be includedwith a support component such as a slide or finger that is removablyattached to the retractor blade 1217. Although the embodimentillustrated in FIGS. 33A and 33B is a retractor blade, the illuminationsystem can be used in conjunction with any number of surgical devices.

Positioning the LEDs at a distance from the optical sensors and usinglight guides to direct light from the illumination sources can provideseveral advantages. For example, illumination sources such as LEDs aresubject to rising temperature during operation. Positioning suchillumination sources in portions of the surgical device, which will beintroduced into a patient's body, presents the danger of increasedtemperature damaging nearby tissue. Fluids also coming in contact withthe LEDs may become heated. In such a configuration, an illuminationsource providing a broad flat light (i.e. having a relatively low levelof illumination per area) may produce similar illumination levels at thesurgical site that a plurality of smaller, hotter illumination sourceswould. Additionally, heat management devices, such as external heatsinks, can disadvantageously add bulk to the illumination source,thereby reducing the working area available to the user. Additionally,as noted above, the use of light guides can allow for mixing of lightfrom the illumination source prior to emission from the light guide. Forexample, in various embodiments the illumination source can comprisemultiple LEDs of different colors. In some embodiments, for example,red, green, blue and possibly amber light emitted from LEDs canpropagate through the light guide and be mixed therein, such thatsubstantially white light is emitted from the light guide and directedtowards the surgical site. By adjusting the properties of the lightguide and the number, intensity, and/or color of the LEDs, theproperties of emitted light from the light guide towards the surgicalsite can be controlled.

Embodiments of Hydraulic Actuator Circuits

Hydraulic actuator circuits can be used in combination with surgicaltools and corresponding imaging systems to perform various functionswithin the surgical systems. For example, without limitation, thehydraulic actuator circuits can power the surgical tools, providecooling to components within the imaging systems, and/or cleancomponents within the system. In some embodiments, using hydraulicactuator circuits and subcomponents thereof to power the surgical toolscan create reduced or minimal electromagnetic interference with trackingtools. For example, electromagnetic fields that may otherwise interferewith electromagnetic tracking need not be inserted into the bodyproximal to the retractor blade and surgical access site and thetracking elements associated with cameras on the retractor blades and/ortracking elements on the surgical tools. Although in some embodimentselectrical motors and devices may be employed to drive hydrauliccomponents, such electrical motors and devices can be located a distancefrom the tracking devices included on the retractor and/or the surgicaltools so as to reduce or eliminate electromagnetic interference. Invarious embodiments, the tracking devices and/or the optical sensors mayinclude shielding such as a casing or shield comprising metal orMu-metal.

In some embodiments, the components of the hydraulic circuit (e.g.,pumps, valves, surgical tools, manifolds) can be made or at leastpartially constructed from non-metal materials (e.g., polymers) and/ornon-ferrous materials. Hydraulic actuator circuits can include manyinexpensive and/or disposable components (e.g., molded components and/orsubcomponents). In some configurations, hydraulic actuator circuits canoperate at low noise volumes compared to pneumatic drills, reducing orminimizing irritation and potential ear damage for the surgeons andmedical staff. The components of the hydraulic actuation circuits can beconstructed from a combination of computer numerical control (CNC) partsand molded parts.

FIG. 34 illustrates an embodiment of a hydraulic actuator circuit 1500.The circuit 1500 preferably includes a hydraulic pressure source. Insome embodiments, the hydraulic pressure source is a fluid interfacechamber 1502. Portions of or the entire fluid interface chamber 1502 canbe disposable and/or non-autoclavable. The fluid interface chamber 1502can be pre-loaded with a hydraulic fluid 1504 (e.g., saline). Asdiscussed in the present disclosure, the term “hydraulic fluid” canrefer to saline (e.g., a physiological saline) and/or to any otherphysiologically compatible fluid (e.g., fluid suited to interaction withthe interior of the body of a patient). Other fluids can be used in thefluid interface chamber 1502 (e.g., distilled water, oil). A pneumaticfluid source 1510 (e.g., a hospital nitrogen system, a nitrogen tank,etc.) can be actuated to exert pressure on the hydraulic fluid 1504within a cavity 1508 of the housing 1506, thereby pressurizing the fluid1504. The pneumatic fluid source 1510 can be controlled via user input(e.g., a treadle, button, dial, etc.) and/or via automatic control(e.g., a pre-programmed computer).

The fluid interface chamber 1502 can output hydraulic fluid 1504 (e.g.,saline) to a tool-powering apparatus. For example, the fluid interfacechamber 1502 can output saline to a saline turbine 1512 configured topower a hydraulic drill or other tool, as further discussed below. Inone embodiment, the fluid interface chamber 1502 can output saline in apulsed manner to cause rotation of the saline turbine 1512 (e.g., viathe impingement of the fluid flow stream with one or more vanes of thesaline turbine 1512). In some embodiments, the saline turbine 1512and/or powered drill/tool are disposable and/or non-autoclavable. Thesaline input into the turbine 1512 can cool the turbine 1512. Theturbine 1512 can include a hydrostatic bearing. In some embodiments,leakage of saline from the turbine 1512 via the hydrostatic bearing orotherwise can irrigate the surgical field. In some embodiments, thesaline turbine 1512 can operate at low noise volumes (e.g., compared topneumatic drills) at high operation speeds (e.g., speeds exceeding40,000 rpm). However, the turbine 1512 can operate at other speeds. Thesaline turbine 1512 can be lightweight, can have a high torque to massratio, and can operate without producing I²R losses, thus reducingheating within the hydraulic actuator circuit 1500. Preferably, thesaline turbine 1512 can be an impulse turbine. In some embodiments, thesaline turbine 1512 is a vane motor, or a Tesla turbine.

In the preferred embodiment, the exhaust fluid 1514 can be output fromthe turbine 1512 to a hydraulic manifold 1516. The manifold 1516 candistribute the exhaust fluid 1514 amongst one or more manifold outputs.For example, a portion of the exhaust fluid 1514 can be directed througha first manifold outlet 1517 to be used to wash optics such as imagingoptics included in cameras on the retractor blades or surgical tools,for example, as discussed above. The manifold 1516 can include one ormore pumps and/or valves. For example, fluid directed toward the firstmanifold outlet 1517 can be manipulated by a linear actuator compressingan elastomeric valve or a multi-lobed cam on a brushless DC motor or aproportional solenoid array to produce tube compression and/orcompression of blisters on the manifold 1516 to produce optics washingpulses. In some embodiments, air or other gases can be input into themanifold 1516 to dry the optics after washing. The optics can besterilized before, after, or during washing. A portion of the fluid 1514can be directed through a second manifold outlet 1518 to cool LEDs usedto provide illumination to the surgical site to facilitate imaging bycameras on the retractor and or surgical tools and viewing by thesurgeon and a portion of the fluid 1514 can be directed through a thirdmanifold outlet 1519 toward hydraulics for a power Kerrison or rongeur.The manifold 1516 can include more than or less than three manifoldoutlets depending on the specific application of the hydraulic circuit1500. In one embodiment, the manifold 1516 can include one or morevalves (not shown) that can be selectively actuated to direct exhaustfluid 1514 to one or more manifold outlets, while not directing exhaustfluid to other manifold outlets. For example, the valves can beselectively actuated (e.g., via a controller based on user inputs orbased on an automatic control algorithm implemented by the controller)so that exhaust flow is directed to the first and second manifoldoutlets 1517, 1518, but not the third manifold outlet 1519, or so thatexhaust flow is directed to the third manifold outlet 1519, but not thefirst and second manifold outlets 1517, 1518. The manifold may compriseflexible plastic and may be ultrasonically welded or adhesive bonded tomake channels with edge connectors for tubing in some embodiments. FIG.35A illustrates an embodiment of a hydraulic circuit 1520 which includesa pump 1522. The pump 1522 can be a roller pump configured to pressurizeexhaust fluid 1514 from the turbine 1512. In some embodiments, a rollerpump can reduce the overall envelope size of the system. The pump 1522can include a roller 1524 configured to compress a flexible tube 1526 ina rotational manner such that fluid entering the pump 1522 via the pumpinlet 1528 is compressed and/or accelerated through the flexible tube1526 to the pump outlet 1530 as the roller 1524 rotates. The flexibletube 1526 and/or other components of the pump 1522 can be disposable.Pressurized fluid exiting the pump outlet 1530 can be directed to thefluid interface chamber 1502 and/or to the turbine 1512. In someembodiments, a bi-directional valve is located between the pump outlet1530 and the fluid interface chamber 1502 and turbine 1512. Such a valvecan be manually or automatically controlled (e.g., via a computercontroller) to direct flow to the fluid interface chamber 1502 and/or tothe turbine 1512. In some embodiments, the hydraulic circuit 1520 orportions thereof can be configured be releasably installed within acassette assembly. The cassette assembly can be disposable and canprovide a condensed volume in which portions of or the entire hydrauliccircuit 1520. Similar or identical cassette assemblies can be used inconjunction with the hydraulic circuits described herein.

According to some variants, as illustrated in FIG. 35B, a hydrauliccircuit 1540 can include a proportioning valve 1534 configured toautomatically direct a desired proportion of the hydraulic fluid outputby the fluid interface chamber 1502 directly to one or both of theturbine 1512 and the hydraulic manifold 1516. The proportional valve1534 can be inexpensive and/or disposable. In some embodiments, theproportional valve 1534 is non-autoclavable. The proportioning valve1534 can be driven by a proportional solenoid or non-commutated linearmotor (e.g., with either moving a coil or moving magnet). FIG. 35Cillustrates a hydraulic circuit 1580 that includes a bellows actuator1536 for use as a hydraulic pressure source. The bellows actuator 1536can include a bellows (e.g., a metal bellows) actuated at least in partby a linear actuator 1542. The linear actuator 1542 can be controlled bya motor 1544 (e.g., a brushless motor) or other controller device. Thebellows actuator 1536 can perform many or all of the functionspreviously described with respect to the fluid interface chamber 1502.For example, the bellows actuator 1536 can provide pressurized hydraulicfluid (e.g., saline) to the proportioning valve 1534 and/or turbine1512. The bellows actuator 1536 can include a return spring that biasesthe bellows 1536 to an expanded position. In one embodiment, acontroller (e.g., a computer controller) can control the operation ofthe motor 1544 (e.g., based on user input, such actuation of a footpedal, push button, lever, voice command, etc.), thereby controlling theoperation of the bellows actuator 1536 to provide a desired fluid flowand/or pressure to the proportioning valve 1534.

The hydraulic circuit 1580 can include a second hydraulic pressuresource. For example, the circuit 1580 can include an IV bag pressurizedfluid source 1546. The IV bag source 1546 can output fluid to adirectional valve 1548. The directional valve 1548 can be constructedinexpensively and/or can be disposable. The directional valve 1548directs the output fluid from the IV bag source to a pump 1522 and/or toa check valve 1552. The check valve 1552 can be configured to permitfluid flow from the directional valve 1548 to the bellows actuator 1536while preventing fluid flow through the check valve 1552 from the bellowactuator 1536 to the directional valve 1548. Fluid directed to the pump1522 can be pressurized within the pump 1522 and output to the hydraulicmanifold 1516. In some embodiments, the bellows actuator 1536 and fluidinterface chamber 1502 are interchangeable structurally and/orfunctionally within the hydraulic circuits 1500, 1520, 1540, 1580.

FIG. 36A illustrates an embodiment of an LED cooling system 1600. Thecooling system 1600 can include a pump 1522 that is similar or identicalto the pump 1522 described above. The system 1600 can include a fluidinput 1610 via a vent line for saline priming. A pinch valve 1608 (e.g.,a solenoid pinch valve) or other valve can be positioned at or near thefluid input 1610. The pinch valve 1608 can be configured to selectivelyinhibit fluid flow into the LED cooling system 1600. The pinch valve1608 can be manually and/or automatically controlled (e.g., via anelectronic or computer controller).

The cooling system 1600 can be used to cool one or more LEDs 1602 usedto provide illumination to the surgical site to facilitate imaging bycameras on the retractor and or surgical tools and viewing by thesurgeon. In this example, 6 LEDs 1602 are shown. Fewer or more LEDs 1602are possible. The fluid (e.g., saline) received into the system 1600 viathe input 1610 (e.g., received via the manifold described above) can bepassed through, over, and/or past the LEDs 1602 to cool the LEDs 1602.The pump 1522 can be used to pump the fluid past the LEDs 1602. In someembodiments, the pump 1522 pumps the fluid back to the hydraulicactuator circuits and/or to an optics washing circuit via a coolingsystem outlet 1612. A second pinch valve 1609 can be positioned withinthe cooling system 1600 to selectively inhibit the fluid output from thefirst pinch valve 1608 from accessing the pump 1522 without passingby/through the LEDs 1602. Accordingly, the cooling of the LED provide bythe hydraulic system can be switched on or off. The pinch valves 1608,1609 can be configured in some embodiments, so that a portion of thefluid that flows into the LED cooling system 1600 can exit the system1600 via the cooling system outlet 1612 without passing through the pump1522 or past the LEDs 1602.

In some embodiments, as illustrated in FIG. 36B, the cooling system 1600includes hydraulic pressure source. The hydraulic pressure source canbe, for example, a saline bag pressure compression system 1630. Thecompression system 1630 can include a bag 1632 pre-charged withhydraulic fluid (e.g., saline) and disposed in a housing, for examplevia fluid flow from fluid input 1610. The compression system 1630 canhave a driven compressing element 1634 (e.g., a plate, piston)configured to compress the bag 1632. The compressing element 1634 can bedriven by a motor or other actuator (e.g., other electrical, mechanicalor pneumatic actuator). For example, the actuator can be a motor 1638configured to drive a Acme or preferably ball screw 1636. The screw 1636can couple with the compressing element 1634 and can move thecompressing element 1634 toward and/or away from the bag 1632 upon inputfrom the motor 1638. The compression system 1630 or other hydraulicpressure source can provide pressurized hydraulic fluid to the coolingsystem 1600. In some embodiments, a portion of the fluid output from thecompression system 1630 is directed to the hydraulic actuator circuitsand/or to an optics washing circuit via the cooling system outlet 1612.The pinch valve 1608, 1609 can be configured to selectively inhibitaccess to the fluid input 1610 by the fluid output from the compressionsystem 1630.

In some embodiments, the hydraulic pressure source of the hydraulicpressure circuits and/or LED cooling systems can be a rolling edgediaphragm, a syringe (e.g., a disposable syringe) or any other suitablehydraulic pressure source or combination or pressure sources (e.g., asyringe in combination with bellows). The hydraulic pressure source(s)can be free from valves.

As illustrated in FIG. 37, a hydraulic pressure circuit 2000 for usewith surgical tools or otherwise can include a fluid source 2010. Thefluid source 2010 can include a fluid reservoir 2012 (e.g., an IV bag orother fluid container). The fluid source 2010 can be gravity-driven. Insome embodiments, the hydraulic pressure circuit 2000 includes a fluidsource actuator configured to force fluid from the fluid source 2010. Insome embodiments, the fluid source 2010 includes a hydraulic pump 2014(e.g. a roller (peristaltic) pump, a linear actuator pump, a gear pump,a radial piston pump, a screw pump, and/or an axial piston pump) fluidlyconnected to the reservoir 2012. The hydraulic pump 2014 can beconfigured to force fluid out of and/or into the fluid reservoir 2012.One or more feedback sensors (e.g., pressure transducers 2018) can beconnected to the fluid reservoir 2012 to monitor the fluid pressurewithin the fluid reservoir 2012 and/or within the fluid lines connectedto the reservoir 2012.

The hydraulic pressure circuit 2000 can include a hydraulic manifold2020. In some embodiments, the hydraulic manifold 2020 is housed atleast partially within a cassette assembly, an example of which will bediscussed below. The hydraulic manifold 2020 can include one or moreexternal connectors 2002. The external connectors 2002 can be configuredto facilitate fluid communication between the hydraulic manifold 2020and other components of the hydraulic circuit 2000. For example, anexternal connector 2002 can provide a fluid interface between thehydraulic manifold 2020 and one or more fluid lines of the fluid source2010.

In some embodiments, the external connectors 2002 define one or morefluid inlets and/or fluid outlets into and out of the hydraulic manifold2020. For example, the external connector 2002 configured to connect tothe fluid source 2010 can define a fluid inlet into the hydraulicmanifold 2020. The hydraulic manifold 2020 can include one or moreexternal connectors 2002 configured to facilitate fluid connectionbetween the hydraulic manifold 2020 and a tool assembly 2080. Forexample, the hydraulic manifold 2020 can include a tool fluid outlet anda tool fluid inlet to facilitate fluid transfer to and from the toolassembly 2080. In some cases, a long flexible tube 2084 (e.g., a sevenfoot flexible tube) can be used to fluidly connect the hydraulicmanifold 2020 to the tool assembly 2080. In some embodiments, thehydraulic manifold 2020 includes external connectors 2002 configured toform one or more LED fluid inlets and one or more LED fluid outlets tofacilitate fluid transfer between the hydraulic manifold 2020 and an LEDassembly 2060. In some embodiments, the LED assembly 2060 can be fluidlyconnected to the fluid source 2010, with or without the use of externalconnectors on the LED assembly 2060 and the fluid source 2010.

The external connectors 2002 can be configured to reduce the time andeffort associated with fluidly connecting and/or disconnecting thecomponents of the hydraulic pressure circuit 2000 (e.g., the fluidsource 2010, the hydraulic manifold 2020, the LED assembly 2060, and/orthe tool assembly 2080) to each other. For example, the connectors 2002can comprise tapered connectors onto which tubing (e.g., flexibleplastic tubing) can be pressure-fit. In some embodiments, the connectors2002 can comprise male and/or female luer connectors (e.g.,ANSI-compliant connector interfaces) configured to connect with acorresponding connector on a fluid conduit. In some embodiments, theconnectors 2002 include internal and/or external threading configured toreleasably engage with external and/or internal threading on a fluidconduit.

According to some variants, one or more valves 2032 can be positioned onor within the hydraulic manifold 2020 to control and/or limit the fluidflow through one or more fluid conduits within the hydraulic manifold2020. The one or more valves 2032 can be, for example, two-way twoposition valves 2032 (e.g., pinch valves). In some embodiments, thehydraulic manifold 2020 can include one or more proportioning valves2034 (e.g., proportional two-way two position valves, diaphragm-typevalves, and/or spindle valves) configured to control and/or limit fluidflow through one or more fluid conduits within the hydraulic manifold2020. In certain cases, a valve 2032, 2034 is positioned on each fluidchannel within the hydraulic manifold 2020 (e.g., FIG. 37). The valve2032, 2034 can act as on/off valves to provide pulsed flow through oneor more of the fluid conduits in the hydraulic manifold 2020.

In some embodiments, the hydraulic manifold 2020 includes one or morefluid actuators. For example, the hydraulic manifold 2020 can includeone or more bellows actuators 2022. Other types of fluid actuators suchas master-slave balloon pumps, axial pistons, and peristaltic pumps canalso be used in addition to or instead of the bellows actuators 2022 topressurize hydraulic fluid within the hydraulic pressure circuit 2000.Although the following discussion will refer to bellows actuators 2022as the fluid actuator, the concepts disclosed herein could also apply toother types of fluid actuators disclosed above. The bellows actuators2022 can be configured to increase fluid pressure within the fluidconduits of the hydraulic manifold 2020. The bellows actuators 2022 caninclude a bellows 2028. One or more linear actuators 2026 can beconfigured to compress and decompress the bellows 2028. The one or morelinear actuators 2026 can be manipulated by one or more motors 2024(e.g., brushless motors, stepper motors).

Fluid can be provided to the bellows actuators 2022 from the fluidsource 2010. One or more valves 2032, 2034 can be positioned in thefluid paths between the fluid source 2010 and the bellows actuators2022. The bellows actuators 2022 can output high pressure fluid to thetool assembly 2080 and/or to the LED assembly 2060. In some embodiments,the hydraulic manifold 2020 includes two or more bellows actuators 2022.In some cases, the use of two or more bellows actuators 2022 can providecontinual pressurization of fluid travelling from the bellows actuators2022 to the tool assembly 2080 and/or to the LED assembly 2060. Forexample, the compression strokes of the linear actuator 2026 of thebellows actuators 2022 can be timed such that, at a given moment in theoperation of the hydraulic pressure circuit 2000, at least one of thelinear actuators 2026 is operating in a compression stroke.

Pressurized fluid output from the bellows actuators 2022 to the toolassembly 2080 can pass through a proportioning valve 2034. Theproportioning valve 2034 can control the amount of fluid and/or thepressure of the fluid that passes from the bellows actuators 2022 to thetool assembly 2080. In some cases, the proportioning valve 2034 cancontrol the operating speed and/or operating power of the tool assembly2080. As will be explained in more detail below, pressurized fluid fromthe bellows actuators 2022 can be passed through a nozzle 2086 andimpeded upon a turbine or other drive source for a tool 2082 (e.g., ahydraulic drill or other surgical tool). In some embodiments, the nozzle2086 intensifies the velocity of the pressurized fluid. At least aportion of the fluid used to drive the tool 2082 can be collected and/orredirected back to the bellows actuators 2022 in order to reenergize thefluid. In some embodiments, a hydrophobic filter 2088 can be positionedin a branch of the fluid channel through which fluid returning from thetool assembly 2080 to the bellow actuators 2022 flows. The hydrophobicfilter 2088 can allow air or other gases from the fluid channel to exitthe hydraulic pressure circuit 2000 while preventing the hydraulic fluid(e.g., the water, saline, and/or oil) from exiting the hydraulicpressure circuit 2000. In some embodiments, a one-way valve can bepositioned in the return fluid channel between the tool assembly 2080and the bellow actuators 2022 to inhibit air or other gases fromentering the hydraulic pressure circuit 2000 through the hydrophobicfilter 2088.

In some embodiments, a portion of the fluid exiting the fluid source2010 can be directed to the LED assembly 2060. This portion of fluid canbe directed through the hydraulic manifold 2020 before it reaches theLED assembly 2060. This portion of fluid can bathe the LEDs 2062 and canprovide conductive and/or convective cooling for the LEDs 2062. In someembodiments, the LED assembly 2060 includes a roller pump or othersource of fluid pressurization that pulls fluid onto and around the LEDs2062 from the hydraulic manifold 2020 and/or from the fluid source 2010.After passing over/around the LEDs 2062, the cooling fluid can bedirected via a fluid conduit back to the fluid source 2010, asillustrated in FIG. 37.

In some embodiments, a portion of the fluid exiting the bellowsactuators 2022 is directed to one or more nozzles 2066 in or on the LEDassembly 2060. An on/off valve can be positioned on or around the fluidconduit between the bellows actuators 2022 and the LED assembly 2060.The on/off valve can be configured to provide a pulsed pressure forwashing the LEDs 2062. The fluid used to wash the LEDs 2062 can beredirected to the fluid source 2010.

In some embodiments, the pressure and pulse rate of the fluid washingand air drying of the LEDs 2062 can be further controlled through theaddition of a check valve near the nozzles 2066. In some embodiments,when a pressure differential across the check valve (e.g., the ratio ofthe upstream fluid pressure to the downstream fluid pressure) reaches apredetermined level, the check valve can open to release a strongerpulse and higher pressure of fluid for washing the LEDs than would beachieved with the pulsing regulated by valves at the cassette. The checkvalve can be a duck billed valve, a diaphragm check valve, lift checkvalve, or any other check valve known in the art. In some embodiments, aT connector can be used near the nozzle connecting the air line and thesaline line to the nozzle, thereby allowing for the air and saline toshare the same nozzle. Additionally, in some embodiments, the two salinelines from the fluid source and/or pistons can be connected by a Tconnector to connect the two fluid lines to the nozzle. A check valvecan be positioned near the nozzle to facilitate a pressure buildup onthe upstream side of the check valve. In some embodiments, such abuildup of pressure can release a high pressure pulse of fluid forwashing the LEDs. Alternatively, in some embodiments, a T connector canbe used to connect a fluid line, from the fluid source and/or thepistons, and an air line. The air line can contain a high pressure andlow pressure air source. The low pressure combined with the fluid linecan exert a force to open the check valve near the nozzle. In someembodiments, a subsequent higher pressure blows air through the nozzleto allow for drying of the LEDs through the same nozzle as washing. Insome embodiments, the pulse of fluid for washing the LEDs can have apressure ranging from about 60 psi to about 125 psi.

Additionally, in some embodiments, the LED assembly 2060 can be cooledby a refrigeration system. In some embodiments, the refrigeration systemcan comprise a micro miniature refrigerator (Joule Thomson effect)cooler to circulating saline and cool the LEDs 2062. The temperaturechange in the compressed gas system when the gas or fluid is forcedthrough a valve can allow for the cooling of the gases or fluids if keptinsulated so that no heat is exchanged with the environment.

FIG. 38 illustrates an embodiment of a hydraulic pressure circuit 2100.Some numerical references to components in FIG. 38 are the same as orsimilar to those previously described for the hydraulic pressure circuit2000 (e.g. hydraulic manifold 2120 v. hydraulic manifold 2020). It is tobe understood that the components can be the same in function or aresimilar in function to previously-described components. The hydraulicpressure circuit 2100 of FIG. 38 shows certain variations to thehydraulic pressure circuit 2000 of FIG. 37.

In some embodiments, the hydraulic pressure circuit 2100 includes an airpump 2190. The air pump 2190 can be configured to provide pressurizedair, via a fluid conduit, to the nozzles 2166 of the LED assembly 2160.In some embodiments, the pressurized air can be used to dry the LEDs2162 before or after washing the LEDs. A filter 2192 (e.g., ahydrophobic and/or antimicrobial filter) can be positioned between theair pump 2190 and the nozzles 2166.

The hydraulic pressure circuit 2100 can include a balloon 2170. Theballoon 2170 can be used as a slave actuator to power a tool (e.g., aKerrison). For example, the balloon 2170 can be used to power tools thatrequire high pressure (e.g., 100-150 psi) actuators. In someembodiments, as discussed below, one or more tools are actuated viapneumatic actuators.

FIG. 39 illustrates an embodiment of a hydraulic pressure circuit 2200.Some numerical references to components in FIG. 39 are the same as orsimilar to those previously described for the hydraulic pressure circuit2100 (e.g. hydraulic manifold 2220 v. hydraulic manifold 2120). It is tobe understood that the components can be the same in function or aresimilar in function to previously-described components. The hydraulicpressure circuit 2200 of FIG. 39 shows certain variations to thehydraulic pressure circuit 2100 of FIG. 38.

In some cases, the hydraulic pressure circuit 2200 can include an airaccumulator 2294 located between the air pump 2290 and the nozzles 2066of the LED assembly 2060. The air accumulator 2294 can provide apre-loaded source of air for use in drying the LEDs 2266 and can helpreduce lag between the operation of the air pump 2290 and the deliveryof air to the nozzles 2066.

A hydraulic pump 2236 (e.g. a roller (peristaltic) pump, a linearactuator pump, a gear pump, a radial piston pump, a screw pump, and/oran axial piston pump) can be positioned within the hydraulic manifold2220. The hydraulic pump 2236 can be positioned and configured toincrease the fluid pressure and/or fluid velocity of fluid output fromthe fluid source 2210. In some embodiments, the hydraulic pump 2236pulls fluid from the fluid source 2210. Fluid from the hydraulic pump2236 can be directed toward the bellows actuators 2222 and/or toward theLED assembly 2260. In some embodiments, positioning the hydraulic pump2236 within the hydraulic manifold 2220 can eliminate the need for anurse or other practitioner to handle the hydraulic pump 2236 duringassembly and/or disassembly of the hydraulic pressure circuit 2200.

FIG. 40 illustrates an embodiment of a hydraulic pressure circuit 2300.Some numerical references to components in FIG. 40 are the same as orsimilar to those previously described for the hydraulic pressure circuit2200 (e.g. hydraulic manifold 2320 v. hydraulic manifold 2220). It is tobe understood that the components can be the same in function or aresimilar in function to previously-described components. The hydraulicpressure circuit 2300 of FIG. 40 shows certain variations to thehydraulic pressure circuit 2200 of FIG. 39.

The hydraulic pressure circuit 2300 can include a second hydraulic pump2338 (e.g. a roller (peristaltic) pump, a linear actuator pump, a gearpump, a radial piston pump, a screw pump, and/or an axial piston pump).The second hydraulic pump 2338 can be positioned on or within thehydraulic manifold 2320. In some embodiments, the second hydraulic pump2338 is positioned on the fluid path through which the fluid exiting thetool assembly 2380 is directed as the exiting fluid is redirected to thebellows actuators 2322. The second hydraulic pump 2338 can be configuredto increase the pressure and/or velocity of said exiting fluid. In someembodiments, the second hydraulic pump 2338 pulls fluid from the toolassembly 2380 after the fluid has been used to power the turbine orother power source for the tool 2382.

FIG. 41 illustrates an embodiment of a hydraulic pressure circuit 2400.Some numerical references to components in FIG. 41 are the same as orsimilar to those previously described for the hydraulic pressure circuit2300 (e.g. hydraulic manifold 2420 v. hydraulic manifold 2320). It is tobe understood that the components can be the same in function or aresimilar in function to previously-described components. The hydraulicpressure circuit 2400 of FIG. 41 shows certain variations to thehydraulic pressure circuit 2300 of FIG. 40.

In some embodiments, the hydraulic pressure circuit 2400 includes asecond tool 2486. The second tool 2486 can be, for example, ahydraulically-driven Kerrison. In some embodiments, the second tool 2486is driven by fluid by a bellows actuators 2422. The hydraulic manifold2420 can include a proportioning valve 2434 positioned on the fluid pathbetween the bellows actuators 2422 and the second tool 2486. Thisproportioning valve 2434 can control the amount of fluid and/or thepressure of the fluid that passes from the bellows actuators 2422 to thesecond tool 2486. In some cases, the proportioning valve 2434 cancontrol the operating speed and/or operating power of the second tool2486. Additional proportioning valves 2434 can be positioned on thefluid paths of the fluid returning from the tool 2482 to the bellowsactuators 2422. These proportioning valves 2434 can be configured tocontrol the rate at which fluid is returned to the bellows actuators2422 from the tool 2484. In some embodiments, one or more of the valves2432 in the hydraulic manifold 2420 can be interchangeable with aproportioning valve 2434.

In some embodiments, the hydraulic pressure circuit 2400 includes apneumatic tool assembly 2450. The pneumatic tool assembly 2450 caninclude a pneumatic tool 2453 (e.g., micro-scissors, micro-forceps). Thepneumatic tool 2453 can be powered by the air pump 2490. In someembodiments, the pneumatic tool assembly 2450 includes a pneumaticmodule 2451 which can include a three-way valve 2456 (e.g., a three-way,two position valve) positioned between the air pump 2490 and thepneumatic tool 2453. The three-way valve 2456 can be configured toselectively direct air or other gases from the air pump 2490 to thepneumatic tool 2453. In some embodiments, the pneumatic tool assembly2453 includes a fluid conduit extending between the pneumatic tool 2453and the three-way valve 2456 and configured to facilitate passage of airor other gases from the pneumatic tool 2453 to the three-way valve 2456.

FIG. 42 illustrates an embodiment of a hydraulic pressure circuit 2500.Some numerical references to components in FIG. 42 are the same as orsimilar to those previously described for the hydraulic pressure circuit2400 (e.g. hydraulic manifold 2520 v. hydraulic manifold 2420). It is tobe understood that the components can be the same in function or aresimilar in function to previously-described components. The hydraulicpressure circuit 2500 of FIG. 42 shows certain variations to thehydraulic pressure circuit 2400 of FIG. 41.

In some embodiments, the pneumatic module 2551 can include an airaccumulator 2558. The air accumulator 2558 can be positioned along thefluid path between the air pump 2590 and the nozzles 2566 of the LEDassembly 2560. The air accumulator 2558 can be configured to reduce lagbetween the production of pressurized gas by the air pump 2590 anddelivery of said pressurized gas to the nozzles 2566. Positioning theair accumulator 2558 outside the hydraulic manifold 2520 can simplifythe structure and design of the hydraulic manifold 2520 unit (e.g., thecassette in which the hydraulic manifold 2520 may be housed).

FIG. 43 illustrates an embodiment of a hydraulic pressure circuit 2600.Some numerical references to components in FIG. 43 are the same as orsimilar to those previously described for the hydraulic pressure circuit2500 (e.g. hydraulic manifold 2620 v. hydraulic manifold 2520). It is tobe understood that the components can be the same in function or aresimilar in function to previously-described components. The hydraulicpressure circuit 2600 of FIG. 43 shows certain variations to thehydraulic pressure circuit 2500 of FIG. 42.

In some embodiments, the air accumulator 2658 can be positioned on or inthe fluid path between the air pump 2690 and the pneumatic tool 2653.The air accumulator 2690 can reduce the lag between the production ofpressurized gas from the air pump 2658 and the delivery of saidpressurized gas to the pneumatic tool 2653 and/or to the nozzles 2666 ofthe LED assembly 2660. Various pneumatic indicators 2655 can bepositioned around and/or within the pneumatic assembly 2650 to providevisual indication of the status of various pneumatic components withinthe pneumatic assembly 2650.

In some cases, the second tool 2686 can be configured to redirect atleast a portion of the fluid used to power the second tool 2686 back tothe hydraulic manifold 2620. For example, a fluid line can extend fromthe second tool 2686 to the hydraulic manifold 2620 such that fluidexhausted from the second tool 2686 can be directed to the bellowsactuators 2622 to reenergize said exhausted fluid.

The hydraulic manifold 2620 can include a fluid-air separator 2637. Theair separator 2637 can be configured to remove air and other gases fromthe fluid line onto which the air separator 2637 is installed. Forexample, the fluid-air separator 2637 can be positioned and configuredto remove air and other gases from the fluid that is directed toward thebellows actuators 2622 from the tool 2682 and/or from the second tool2686. Such air-gas mixtures can result from actively scavenging lowvelocity hydraulic fluid and air from the tool 2682 and/or second tool2686 via annular vents around the turbines of the tools or otherwiseafter the hydraulic fluid is used to power the tool 2682 and/or secondtool. Removal of gases from the fluid lines of the hydraulic manifold2620 can improve the performance of the hydraulic components of thehydraulic pressure system 2600 (e.g., the tools 2682, 2686). In someembodiments, the hydraulic manifold 2620 includes a pressurecompensation source 2639 (e.g., a master-slave balloon) configured toregulate the pressure in one or more of the fluid lines of the hydraulicmanifold 2620.

In some embodiments, one or more of the valves within the hydraulicpressure circuits described above can be a consumable valve (e.g., thevalve can be one-time use and/or disposable). For example, one or moreof the valves within the hydraulic manifolds can be a consumable checkvalve configured to be discarded after use. Use of disposable valve canreduce the part and maintenance costs associated with the hydraulicpressure circuits. For example, the reusable valves or other reusablecomponents of the hydraulic manifold (e.g., valves, housings, fluidlines, pumps) could be constructed from polymers, elastomers, and/orother low-cost materials.

FIG. 43A illustrates an embodiment of a hydraulic pressure circuit 2700.Some numerical references to components in FIG. 43A are the same as orsimilar to those previously described for the hydraulic pressure circuit2600 (e.g. hydraulic manifold 2720 v. hydraulic manifold 2620). It is tobe understood that the components can be the same in function or aresimilar in function to previously-described components. The hydraulicpressure circuit 2700 of FIG. 43A shows certain variations to thehydraulic pressure circuit 2600 of FIG. 43.

In some embodiments, the hydraulic pressure circuit 2700 can include apneumatic pressure assembly 2765. The pneumatic pressure assembly 2765can function as a pressure source for the hydraulic fluid of thehydraulic pressure circuit 2700. In some embodiments, the pneumaticpressure assembly 2765 is used in combination with mechanical actuators(e.g., bellows similar to or the same as bellows 2628) to pressurize thehydraulic fluid within the hydraulic pressure circuit 2700. In someembodiments, the pneumatic pressure assembly 2765 is used instead ofsuch mechanical actuators to pressurize the hydraulic fluid within thehydraulic pressure circuit 2700.

The pneumatic pressure assembly 2765 can include one or more pneumaticactuators. The pneumatic actuators can be configured to pressurize airor other gases within the pneumatic pressure assembly 2765. For example,the pneumatic pressure assembly 2765 could include one or more linearactuators 2726, 2726′ driven by one or more motors 2724, 2724′. Thelinear actuators 2726, 2726′ can be used to increase and decrease thepressure of a pneumatic fluid within a pneumatic actuator chamber 2723,2723′. For example, the linear actuators 2726, 2726′ can be used toincrease and decrease the size (e.g., the volume) of the pneumaticactuator chambers 2723, 2723′ to effect changes in pressure for thepneumatic fluid within the actuator chambers 2723, 2723′.

In some embodiments, the pneumatic pressure assembly 2765 includes oneor more pneumatic pressure sources. The one or more pneumatic pressuresources can be in fluid communication with the actuator chambers 2723,2723′. For example, the pneumatic pressure assembly 2765 can include apump 2796. The pump 2796 can be configured to provide pressurizedpneumatic fluid (e.g., air and/or other gases) to the actuator volumes2723, 2723′. The pump 2796 can be configured to pressurize the pneumaticfluid to pressures above atmospheric (e.g., 60 psi). In someconfigurations, one or more valves 2727, 2727′ can be positioned in thefluid path(s) between the pump 2796 and the chambers 2723, 2723′. Insome embodiments, the use of the pump 2796 can reduce or eliminate aneed to connect the pneumatic pressure assembly 2765 and/or hydraulicpressure system 2700 to a hospital compressed air system. The valves2727, 2727′ can be configured to selectively allow and/or constrictfluid communication between the pump 2796 and the chambers 2723, 2723′.In some embodiments, the valve 2727, 2727′ are three way, two positionvalves. Many different types of valves are possible (e.g., one-way checkvalves, pincher valves, solenoid valves, etc.).

The pneumatic actuator chambers 2723, 2723′ can be in fluidcommunication with secondary pneumatic actuator chambers 2723 a, 2723a′. In some embodiments, one or more valves 2729, 2729′ can bepositioned in the fluid path(s) between the actuator chambers 2723,2723′ and the secondary pneumatic actuator chambers 2723 a, 2723 a′ Thevalves 2729, 2729′ can be configured to selectively allow and/orconstrict fluid communication between the actuator chambers 2723, 2723′and the secondary pneumatic actuator chambers 2723 a, 2723 a. In someembodiments, the valve 2729, 2729′ are three way, two position valves.Many different types of valves are possible (e.g., one-way check valves,pincher valves, solenoid valves, etc.).

The secondary pneumatic actuator chambers 2723 a, 2723 a′ can bepositioned at least partially within the hydraulic manifold 2720.Pressure within chambers 2723 a, 2723 a′ can exert force upon floatingpiston heads 2726 a, 2726 a′. In some embodiments, the secondarychambers 2723 a, 2723 a′ do not include floating piston heads and thepressurized pneumatic fluid interacts directly with the hydraulic fluidin the chambers 2723 a, 2723 a′. The floating piston heads 2726 a, 2726a′ can define an interface between the pneumatic fluid of the pneumaticpressure assembly 2765 and the hydraulic fluid F of the hydraulicmanifold 2720 and hydraulic pressure circuit 2700. Exertion of forceupon the floating piston heads 2726 a, 2726 a′ by the pneumatic fluidwithin the chambers 2723 a, 2723 a′ can cause the floating piston heads2726 a, 2726 a′ to increase the pressure of the hydraulic fluid F on theside of the floating piston heads 2726 a, 2726 a′ opposite the chambers2723 a, 2723 a′. In embodiments without piston heads, introduction ofpressurized pneumatic fluid into the chambers 2723 a, 2723 a′ canincrease the pressure of the hydraulic fluid F within the chambers 2723a, 2723 a′. In some embodiments, the floating piston heads 2726 a, 2726a′ in combination the pressurized pneumatic fluid within the secondarypneumatic actuator chambers 2723 a, 2723 a′ can perform the same or asimilar function as performed by the bellows actuators described aboveand below. In some embodiments without floating piston heads, thepressurized pneumatic fluid acting directly on the hydraulic fluidwithin the chambers 2723 a, 2723 a′ can perform the same or a similarfunction as performed by the bellows actuators described above andbelow.

In some embodiments, the pneumatic pressure assembly 2765 includes oneor more pneumatic indicators 2725, 2725′, 2725 a, 2725 a′. In someembodiments, pneumatic indicators 2725 a, 2725 a′ are redundant backupindicators in the case of malfunction and/or failure of the indicators2725, 2725′. The indicators can be configured to monitor the pressurewithin the fluid lines connecting the pneumatic actuator chambers 2723,2723′ with the secondary pneumatic actuator chambers 2723 a, 2723 a′.The indicators can be operably coupled (e.g., electrically connected viawired and/or wireless connections) with the motors 2724, 2724′ and/orwith the linear actuators 2726, 2726′. In some embodiments, theindicators 2725, 2725′, 2725 a, 2725 a′ are operably coupled with thepump 2796 and/or other pneumatic pressure source.

According to some configurations, the pneumatic pressure assembly 2765can operate in the following manner. During a compression stroke, thelinear actuator 2726 can be configured to compress the pneumatic fluidwithin the chamber 2723 (e.g., in response to input from the motor 2724)to maintain a predetermined pressure (e.g., 120 psi) within the fluidline connecting the chamber 2723 with the secondary chamber 2723 a. Theindicator 2725 can monitor the pressure within the fluid line and cancommunicate (e.g., via an operable connection such as a wired orwireless electrical connection) with the motor 2724 to ensure that thepressure within the fluid line remains at or above a minimum acceptablepressure.

The valve 2729 can be configured to permit fluid communication betweenthe chamber 2723 and the secondary chamber 2723 a during the compressionstroke of the linear actuator 2726. Compression of the pneumatic fluidwithin the chamber 2723 during the compression stroke can cause anincrease in pressure within the secondary chamber 2723 a. Increase inpressure of the pneumatic fluid in the secondary chamber 2723 a canforce the floating piston 2726 a to (or can directly, in the case of apiston-less embodiment) compress the hydraulic fluid F, thus pressuringthe fluid F for use in the hydraulic pressure circuit.

Upon completion of the compression stroke of the linear actuator 2726,the valve 2729 can cut off fluid communication between the chamber 2723and the secondary chamber 2723 a and allow the secondary chamber 2723 ato vent to ambient. At this point, the pneumatic fluid within thesecondary chamber 2723 a can vent to atmosphere and the hydraulic fluidF provided by the return lines from the tool 2782 and/or second tool2786 and/or from the fluid reservoir 2712 can increase the pressure onthe hydraulic side of the floating piston 2726 a. Such an increase inpressure can cause the piston 2726 a to move and reduce the volume ofthe secondary chamber 2723 a. In embodiments without piston heads in thechambers 2723 a, 2723 a′, the introduction of hydraulic fluid F from thereturn lines from the tool 2782 and/or second tool 2786 and/or from thefluid reservoir 2712 can force the pneumatic fluid out from the chambers2723 a, 2723 a′.

Concurrent with the venting of the secondary chamber 2723 a, the valve2727 can be opened to permit fluid communication between the pump 2796and the chamber 2723. The pump 2796 can provide pneumatic fluid (e.g.,air and/or other gases) at a pressure above ambient (e.g., 60 psi) tohelp facilitate retraction of the linear actuator 2726. In someembodiments, the motor 2724 can act in combination with or instead ofthe pump 2796 to quickly retract the linear actuator 2726. Uponrefilling of the chamber 2723 with pneumatic fluid (e.g., uponcompletion of the expansion stroke of the linear actuator 2726), thevalve 2727 can cut off fluid communication between the pump 2796 and thechamber 2723. In some embodiments, the valve 2727 can be configured tofacilitate venting of the pressurized fluid from the pump 2796 toambient. Although the operation of the pneumatic circuit 2765 has beendescribed in the context of one linear actuator 2726, all or at leastmost of the functions described with regard to linear actuator 2726 andits corresponding valves and chambers equally describe the functioningof linear actuator 2726′ and its corresponding valves and chambers(e.g., chambers 2723′ and 2723 a′, valves 2727′ and 2729′, piston 2726a′). In some embodiments, the use of more than one linear actuator 2726and corresponding valves and chambers can facilitate substantiallycontinual pressurization of the hydraulic fluid F in the hydraulicpressure circuit 2700. For example, while the linear actuator 2726 is inthe compression stroke, the linear actuator 2726′ can be configured tooperate in the expansion stroke such that, upon completion of thecompression stroke of the linear actuator 2726, the compression strokeof the linear actuator 2726′ can begin.

FIG. 43B illustrates an embodiment of a hydraulic pressure circuit 3000.Some numerical references to components in FIG. 43B are the same as orsimilar to those previously described for the hydraulic pressure circuit2700 (e.g. hydraulic manifold 3020 v. hydraulic manifold 2720). It is tobe understood that the components can be the same in function or aresimilar in function to previously-described components. The hydraulicpressure circuit 3000 of FIG. 43B shows certain variations to thehydraulic pressure circuit 2700 of FIG. 43A.

In some embodiments, the fluid lines connecting the various componentsof the pneumatic pressure assembly 3065 can include one or more releasevalves 3067 configured to open if the pressure within a given fluid linerises above a predetermined maximum (e.g., 150 psi). The release valves3067 can be positioned on the fluid lines between the valves 3027, 3027′and the chambers 3023, 3023′. In some embodiments, the fluid lineconnecting the pump 3096 with the chambers 3023, 3023′ can include aventing valve 3061 configured to selectively vent to atmosphere. Such aventing valve 3061 can improve the safe operation of the pneumaticpressure assembly 3065 and can allow the pump 3096 to continue safelyrunning throughout the use of the pneumatic pressure assembly 3065. Insome embodiments, the fluid line connecting the pump 3096 with thechambers 2723, 2723′ can include a venting valve and/or pressure reliefvalve 3069 configured to open and vent to atmosphere upon detection of apredetermined maximum pressure (e.g., 70 psi).

In certain configurations, the pump 3096 can be in fluid communicationwith and supply pressurized pneumatic fluid to the air accumulator 3058of the pneumatic assembly 3050. A pressure regulator 3069 a can bepositioned on the fluid line between the pump 3096 and the accumulator3058 to regulate the pressure within the line. In some embodiments, oneor more pneumatic indicators 3055, 3063 can be positioned on the fluidline between the pump 3096 and the air accumulator 3058. In someembodiments, such an arrangement can eliminate the need for a secondpump 2790 and can simplify the overall design of the hydraulic pressurecircuit 3000. The pneumatic assembly 3050 can include one or more ventvalves 3034 c (e.g., valves configured to vent to atmosphere) configuredto reduce the risk of over-pressurizing the pneumatic assembly 3050. Insome embodiments, a pair of pneumatic indicators (e.g., pressuresensors) 3055 a, 3055 b are positioned on the fluid lines between theair accumulator 3058 and the vent valve 3034 c and between the airaccumulator 3058 and the pneumatic tool 3053. A restriction 3057 (e.g.,an orifice) between the two pneumatic indicators 3055 a, 3055 b can helpto detect an open pneumatic line. In some embodiments, a restriction3057 can help prevent sudden reaction in the pneumatic tool 3053 whenthe vent valve 3034 c is opened to vent the pneumatic assembly 3050.

One or more of the valves (e.g., valves 3027, 3027′, 3029, 3029′, 3034a, 3034 b, 3034 c) in the pneumatic pressure assembly 3065 and/or in thepneumatic assembly 3050 can be spring-driven. For example, the valvescan be biased to the open/venting configuration in the default position.In some embodiments, such a configuration can help reduce the risk ofover-pressurization of the pneumatic pressure assembly 3065 and/or ofthe pneumatic assembly 3050 and can ensure venting of the assemblies3065, 3050 upon shut down of the assemblies 3065, 3050.

One or more of the valves 3034 on the cassette 3020 can be diaphragmvalves formed by an elastomeric element. The valves 3034 can be used tocontrol the first tool 3082 and/or second tool 3086 (e.g., a Kerrisonand/or drill). The valves 3034 can be actuated by non-disposable linearelectromagnetic actuators.

FIG. 43C illustrates an embodiment of a hydraulic pressure circuit 2800.Some numerical references to components in FIG. 43C are the same as orsimilar to those previously described for the hydraulic pressure circuit3000 (e.g. linear actuators 2824, 2824′ v. linear actuators 3024,3024′). It is to be understood that the components can be the same infunction or are similar in function to previously-described components.The hydraulic pressure circuit 2800 of FIG. 43C shows certain variationsto the hydraulic pressure circuit 3000 of FIG. 43B.

For example, the hydraulic pressure circuit 2800 can include a hydraulicpump 2814 (e.g., a peristaltic pump) or other fluid pressurizingcomponent to inhibit fluid backflow into the fluid source 2812 from thesecondary chambers 2823 a, 2823 a′. One way valves 2832 can bepositioned in the fluid paths between the fluid source 2812 and thesecondary chambers 2823 a, 2823 a′ to selectively permit refilling ofone or more of the secondary chambers 2823 a, 2823 a′ from the fluidsource 2812.

The pneumatic pressure assembly 2865 can include two or more linearactuators 2824, 2824′ configured to translate within two or morechambers 2823, 2823′. The fluid lines between the chambers 2823, 2823′and the secondary chambers 2823 a, 2823 a′ can include air inlet valves2827, 2827′. In some embodiments, the inlets to the secondary chambers2823 a, 2823 a′ include three-way solenoid valves 2832 a, 2832 a′configured to selectively allow fluid ingress/egress two and from thesecondary chambers 2823 a, 2823 a′. The three-way solenoid valves 2832a, 2832 a′ can be configured to vent the fluid lines between thechambers 2823, 2823′ and the secondary chambers 2823 a, 2823 a′. Pinchvalves 2832′ or other valves can be used to selectively permit passageof pressurized hydraulic fluid from the secondary chambers 2823 a, 2823a′ to the remaining hydraulic circuit (e.g., via fluid path 2899).

FIG. 43D illustrates an embodiment of a hydraulic pressure circuit 2900.Some numerical references to components in FIG. 43D are the same as orsimilar to those previously described for the hydraulic pressure circuit2800 (e.g. secondary chambers 2923 a, 2923 a′ v. secondary chambers 2823a, 2823 a′). It is to be understood that the components can be the samein function or are similar in function to previously-describedcomponents. The hydraulic pressure circuit 2900 of FIG. 43D showscertain variations to the hydraulic pressure circuit 2800 of FIG. 43C.

For example, the pneumatic pressure assembly 2965 can include a singlepneumatic actuator chamber split into a first pneumatic actuator chamber2923 and a second pneumatic chamber 2923′. A single linear actuator 2924can be transitioned back and forth (e.g., right and left in FIG. 43D) toalternately compress and expand the first and second chambers 2923,2923′. A pump 2996 can be used to provide compressed pneumatic fluid tothe chambers 2923, 2923′ during their respective expansion strokes viavalves 2995, 2995′ (e.g., solenoid valves). In some embodiments, thepneumatic circuit 2965 includes a pressure chamber 2997 configured tostore pre-compressed pneumatic fluid for distribution to the chambers2923, 2923′.

As explained above, portions of or the entire hydraulic manifold 2020,2120, 2220, 2320, 2420, 2520, 2620 (2020 hereinafter for simplicity) canbe housed within a cassette containment apparatus. For example, asillustrated in FIG. 44A, a cassette housing 2021 can include a number ofexternal connectors 2002. One or more valves 2032 (e.g., pinch valves,diaphragm valves, elastomeric valves) can be attached to the cassettehousing 2021. In some embodiments, the valve assembly 2032 on thecassette housing 2021 can have a flexible pad 2041, a cavity 2040, avalve opening 2038, and a pincher or valve stem actuator 2039. The valvestem actuators 2039 can be fixed to a console into or onto which thecassette is connected. The valve stem actuators 2039 can be fixed to aconsole into or onto which the cassette is connected. In someembodiments, the valve stem 2039 can exert a downward force on theflexible pad 2041 thereby compressing the flexible pad 2041 into thecavity 2040 of the cassette housing 2021 above the valve opening 2038and toward the valve opening 2038. The valve opening 2038 can be fullyobstructed, partially obstructed, or fully open at varying amountsdepending on the compression of the flexible pad 2041 by the valve stem2039. The degree of obstruction of the valve opening 2038 by theflexible pad 2041 can regulate, direct, or control the flow of fluidinto the valve. For example, the valve stem 2039 can be used to pulsethe flow of fluid through a valve opening 2038. In some embodiments, thevalve stem 2039 can be used to accelerate fluid flow through a valveopening 2038 (e.g., the valve stem 2039 can constrict the valve opening2038 and act as a nozzle).

In some embodiments, the valve opening 2038 can be obstructed by theflexible pad 2041 and can allow for precise control of the degree andperiod of obstruction by the valve stems 2039 depression onto to theflexible pad. The flexible valve opening can allow for the flow of fluidthrough the valve to be obstructed at such varying degrees or with apulsing flow thereby allowing for a fluid flow through the valve to besustained at a wide range of desired flow rates by controlling thefrequency and distance of depression of the flexible pad 2041 by thevalve stem 2039. For example, the valves 2032 in and on the cassettehousing 2021 can act as diaphragm valves to flex flexible pads on thetop of the cassette housing 2021 and throttle the substantially constanthigh pressure hydraulic fluid supplied from the bellows or otherhydraulic pressure source. In some embodiments, the valve stem 2039 canbe depressed into the flexible pad 2041 by mechanical or hydraulicallydriven forces, or other forces known in the art or described herein. Insome such configurations, pressurized hydraulic fluid can beproportionally directed to the various subsystems and fluid pathways ofthe hydraulic pressure circuit (e.g., the hydraulic tools, LED arrays).In some configurations, the cassette housing 2021 is a substantiallyclosed container. In some embodiments, the cassette housing 2021 has oneor more opened portions.

As illustrated in FIGS. 44B and 44C, the cassette assembly 2021 caninclude a cassette cap portion 2021 a and a cassette body 2021 b. Thecassette cap portion 2021 a can include one or more flexible pads 2041,one or more valves 2032, and one or more cavities 2040. Additionally,the pinchers or valve stem actuators 2039 can be included in thecassette assembly 2021. The cassette body 2021 b can include one or morecylindrical containers configured to house the bellows 2028 of thebellows actuators 2022. In some embodiments, the cassette assembly 2021includes a plurality of tubing sections 2023 configured to connectvarious components and external connectors 2002 to one another. One ormore of the components of the cassette assembly 2021 and/or of thehydraulic manifold 2020 can be consumable. For example, as explainedabove, the valves, fluid lines, body, cap portion, bellows, and/or portsof the cassette assembly 2021 can be consumable. The use of consumableparts can reduce the amount of assembly required hydraulic manifold 2020and can reduce the likelihood of introducing contaminants to thehydraulic pressure circuit 2000 and can reduce or eliminate the cleanupprocess required for the hydraulic pressure circuit 2000 after use.

FIGS. 44D-44F illustrate an embodiment of a cassette assembly configuredto contain the at least a portion of the hydraulic manifold 2720, 3020.The cassette housing 3021 can include one or more connector interfaces3002. The cassette housing 3021 can be a substantially closed container.In some embodiments, the cassette 3021 is at least partially open on oneor more side.

In some embodiments, the cassette housing 3021 includes a cassette capportion 3021 a and a cassette body 3021 b. The cassette cap portion 3021a can define one or more cap fluid channels 3057 a configured to providefluid communication between fluid lines and connectors/ports in and onthe cassette 3021. In some embodiments, the fluid channels can containthe cavity 2040 in which the flexible pads 2041 are depressed into bythe valve stem 2039 as described herein. Fluid channel caps 3059 can bepositioned on top of the cap fluid channels 3057 a to inhibit fluid fromleaking from the channels 3057 a. In some embodiments, the fluid channelcaps can be a flexible pad 2041 used for the valve assembly as describedherein. The fluid channel caps 3059 can be constructed from a flexibleor semi-flexible material (e.g., elastomers, polymers, etc.). Valvesstems 2039 can be used to flex the caps 3059. The valves stems 2039 canbe used to close off or open the connectors/ports in the cassette 3021.In some embodiments, the valve stems 2039 act as diaphragm valves andrestrict the flow paths in the fluid channels 3057 a without completelyclosing off the connectors/ports. In some such configurations, thevalves stems 2039 act as proportional valves to selectively distributehydraulic fluid flow between the various components and subsystems ofthe hydraulic pressure circuit 3000 (e.g., the hydraulic tools, the LEDarray, etc.). One or more flexible and/or rigid tubing sections 3057 canbe used to connect fluid connectors/ports (e.g., connectors interfaces3002) within the cassette 3021. The secondary chambers 3023 a, 3023 a′can, in some embodiments, be housed at least partially within thecassette 3021.

As explained above, the tool 2082, 2182, 2282, 2382, 2482, 2582, 2682(hereinafter 2082 for simplicity) can be driven by a hydraulic turbine.In some embodiments, as illustrated in FIGS. 45A and 45B, a hydraulicturbine 2070 includes a turbine housing 2071. In some cases, at least aportion of a nozzle frame 2072 is housed within the turbine housing2071. In some embodiments, stator vanes can be used in conjunction withand/or in place of the nozzle frame 2072. The nozzle frame 2072 caninclude one or more turbine nozzles 2073. In some embodiments, theturbine nozzles 2073 are positioned in a circumferential array, asillustrated in FIG. 45A. Each of the turbine nozzles 2073 can have anozzle inlet 2074 and a nozzle outlet 2075. In some embodiments, thenozzles 2073 have substantially constant cross-sectional areas fromnozzle inlet 2074 to nozzle outlet 2075 (e.g., drill hole-type nozzles).For example, circular nozzles can be used.

The relative areas of the nozzle inlet 2074 and the nozzle outlet 2075can vary. For example, the nozzle outlet 2075 can have an area that isgreater than or equal to approximately 125% of the area of the nozzleinlet 2074 and/or less than or equal to about 600% of the area of thenozzle inlet 2074. In some embodiments, the area of the nozzle outlet2075 is approximately 300% of the area of the nozzle inlet 2074.

As illustrated in FIG. 45B, the profile of the nozzle 2073 can widenbetween the nozzle inlet 2074 and the nozzle outlet 2075. The rate atwhich the turbine nozzle 2073 widens between the nozzle inlet 2074 andthe nozzle outlet 2075 can vary. For example, the nozzle 2073 can flareout in the direction of the nozzle outlet 2075. In some embodiments, theprofile of the nozzle 2073 narrows between the nozzle inlet 2074 and thenozzle outlet 2075. In some embodiments, the nozzles 2073 havesubstantially constant cross-sectional areas from nozzle inlet 2074 tonozzle outlet 2075 (e.g., drill hole-type nozzles). In some embodiments,the nozzle inlet 2074 can be tapered or flared in such that an openinginto the nozzle inlets 2074 is wider or larger than a midsection of thenozzles 2073.

In some embodiments, hydraulic fluid is directed through the nozzleframe 2072 toward an impeller 2076. The impeller 2076 can include aplurality of impeller blades 2077 around the outer periphery of the hubof the impeller 2076. The impeller blades 2077 can rotate within a bladecavity 2077 a. The impeller 2076 can be integral with or otherwiserotationally coupled with an output shaft 2079 for driving the tool2082, which can be a drill or other rotational tool. The outer diameterof the hub of the impeller 2076 can be smaller than the outside diameterof the array of hydraulic nozzles 2073. For example, the outer diameterof the hub of the impeller 2076 can be greater than or equal toapproximately 15% of the outer diameter of the hydraulic nozzles 2073and/or less than or equal to approximately 75% of the outer diameter ofthe hydraulic nozzles 2073. In some cases, the outer diameter of theimpeller 2076 can be greater than or equal to 0.5 inches and/or lessthan or equal to approximately 1.5 inches. Many variations sizes andrelative sizes of the components of the hydraulic turbine 2070 and itssubcomponents are possible.

In some cases, the impeller blades 2077 are oriented at an angle offsetfrom the central axis of the impeller 2077. The hydraulic nozzles 2073can be configured to turn the flow of hydraulic fluid from an axialdirection A to nozzle direction 2078 as the flow is passed through thenozzle frame 2072 toward the impeller 2076. The nozzle direction 2078can be selected to be at an angle θ_(T) offset from axial A such thatthe nozzle direction 2078 is substantially perpendicular to the faces ofthe impeller blades 2077. The closer nozzle outlets are to the plane ofthe impeller blades 2077 and the more radially-directed the flow fromthe nozzles, the more torque can be imparted upon the impeller blades2077. For example, the nozzle outlets can be positioned close to theimpeller blades 2077 in the axial direction and can direct hydraulicfluid at a highly-radial angle toward impeller blades 2077 whosesurfaces are close to parallel to the rotation of axis of the impeller2076.

In some cases, utilizing a plurality of circumferentially-distributedturbine nozzles 2073 to drive a plurality of impeller blades 2077 canincrease the torque output of impeller 2076 as compared to aconfiguration wherein only one turbine nozzle 2073 is utilized. In somesuch configurations, the outer diameters of the nozzle frame 2072 andimpeller 2076 can smaller than a single-nozzle configuration of equaloutput torque.

In some embodiments, the hydraulic turbine 2070 can be configured tooperate at rotational speeds of 40,000 rpm to 60,000 rpm, though higherand lower rpm values may be possible. The hydraulic turbine 2070 can beconfigured to operate at operating pressures between 70 psi and 190 psi,though greater and lesser operating pressures are possible. In someembodiments, the operating pressure of the hydraulic turbine 2070 isdesigned to be approximately 120 psi.

As illustrated in FIG. 46, an impeller 2076′ can be designed to havebucket-shaped impeller blades 2076′. The bucket-shaped impeller blades2077′ can be oriented at an angle of approximately 45° from the axialdirection A. Many variations of the impeller blade 2077′ angles arepossible. Additionally, many different shapes of blades 2077 arepossible, such as Pelton or Turgo shaped blades.

As illustrated in FIG. 45C, the hydraulic turbine 2070 can be designedto collect the hydraulic fluid that has already impacted the impellerblades 2077, 2077′ (hereinafter 2077 for simplicity). For example, anexhaust angle can be calculated to represent the angle at whichhydraulic fluid reflects off of the impeller blades 2077 after impactwith the impeller blades 2077. One or more vacuum ports 2093 can bepositioned on or in the turbine housing 2071 to extract the fluid F1that is reflected off of the impeller blades 2077 and redirect the fluidF1 into a bypass channel 2095. In some embodiments, the vacuum sourcecan be an external pump (e.g., a peristaltic pump) or the vacuum can bethe result of a Venturi effect created by the diversion of fluid. Forexample, in some embodiments, the vacuum source can be provided bydiverted, high velocity fluid F2 directed to bypass the impeller 2076.In some embodiments, one or more ports 2093 in the hydraulic turbinehousing 2071 (e.g., on the side of the housing closer to the impeller2076 than to the nozzle frame 2072) can create fluid communicationbetween the reflected fluid F1 in the blade cavity 2077A and thediverted high velocity fluid F2 in the bypass channel 2095. The pressuredifferential between the two fluid bodies (e.g., lower pressure in fluidF2 and higher pressure in fluid F1) will pull the reflected fluid F1 outof the housing 2071 and into the diverted fluid path 2095. Removal ofthe reflected fluid from the housing 2071 can increase the performanceof the turbine 2070 by reducing the viscous drag on the impeller fromundiverted fluid F1. For example, the viscous frictional losses thatwould be otherwise incurred from interaction between the reflected fluidF1 and the impeller 2076 and/or output shaft 2079 can be reduced. Thediverted high velocity fluid F2 and scavenged reflected fluid F1 can bediverted back to the cassette 2020 for re-pressurization. In someembodiments, scavenging reflected fluid F1 and diverting it back to thecassette 2020 can reduce the amount of hydraulic fluid (e.g., saline)required to operate the tools and/or other components of the system.

In some embodiments, multiple impellers 2076 (e.g., multiple turbinewheels) can be utilized in the same turbine housing 2071. In some suchembodiments, the overall diameter of the turbine 2070 and/or some of itscomponents can be reduced relative to a single-impeller turbine 2070without sacrificing output torque.

Some instruments such as surgical tools use torque or mechanical forceto translate manual input into tool actuation. For example, a Kerrisonfor bone removal generally includes a handle mechanically coupled to ahead including a stationary portion and a movable portion. When a usersqueezes the handle, the movable portion moves closer to the stationaryportion in a cutting manner (e.g., in a shearing manner), for example toremove bone by trapping the removed bone between the stationary portionand the movable portion (e.g., within a channel between the stationaryportion and the movable portion). Other examples of tools include ananeurysm clipper, a rongeur, forceps, scissors, and the like, althoughmany other hand-operated tools are known to those skilled in the art.Referring again to the Kerrison, the pace and force of the squeezingtranslates to the pace and force of the cutting, and this phenomenon isalso applicable to other hand-operated tools. This translation can bedisadvantageous, for example varying based on each user, being too slowor too fast or having variable speed, lacking force or imparting toomuch force or having variable force, etc. Additionally, periodic use ofsuch manually operated tools (e.g., during a lengthy operation orprocedure) can lead to hand fatigue of the surgeon or user. Manualactuation leads to inadvertent movement of the tool tip.

FIGS. 47A and 47B schematically illustrate an example embodiment of asystem 1700 for hydraulically actuating instruments. The system 1700includes a user interface 1702, a drive system 1704, a pusher member1706 (e.g., piston, plunger), a fluid reservoir 1708, a first inflatableelement 1710, a housing 1712, a fluid conduit 1714, a second inflatableelement 1716, a chamber or housing 1718, a piston 1720, and aninstrument 1722. In some embodiments, a shaft 1726 can extend away fromthe piston 1720 to actuate any number of tools, such as a Kerrison,standard forceps, micro forceps, bipolar forceps, ronguer, clipappliers, scissors, or any other desired tool, as will be described ingreater detail below. The system 1700 allows the user to actuate theinstrument 1722 by operation of the user interface 1702. The userinterface 1702 is not mechanically coupled to the instrument 1722, sothe forces on the user interface 1702 are not necessarily directlytranslated to the forces on the instrument 1722.

Any of the tool embodiments disclosed herein, including any of theKerrison, standard forceps, micro forceps, bipolar forceps, ronguer,clip appliers, or scissors embodiments disclosed herein, can beconfigured to be anatomically designed to fit snugly in a particularorientation relative to a user's hand. Configuring the tools for apredetermined orientation in a user's hand can dictate the orientation(rotational orientation or otherwise) of the tool relative to areference point or surface, such as a ground surface. In thisconfiguration, a CMOS sensor supported by the tool will always beoriented right side up (i.e., in the proper orientation), therebyeliminating the need for PIP image rotation. PIP computation will thenbe translational and scaling only.

Operation of the user interface 1702 sends a signal to the drive system1704. The user interface 1702 may include a proportional foot pedal, apush button, lever and the like. In some embodiments, the user interface1702 is analog, where different levels of operation of the userinterface 1702 cause different responses by the drive system 1704. Insome embodiments, the user interface 1702 is digital, where the drivesystem 1704 responds the same regardless of the level of input on theuser interface 1702. In some embodiments, the user interface 1702includes a single direction, for example a pedal that may be operatedonly forward. In some embodiments, the user interface 1702 includes aplurality of directions, for example a pedal that may be operatedforward or backward. The user interface 1702 may be biased (e.g., by aresilient force such as a spring) to a particular orientation, forexample to a resting point opposite the single direction or to a pointbetween a plurality of directions.

In response to the signal sent from the user interface 1702, the drivesystem 1704, which may for example comprise a linear actuator, drives apushing member or plunger 1706 by way of a proportional solenoid, Acemor ball screw, and the like. In some embodiments, the drive system 1704converts random forces on the user interface 1702 into uniform, known,and/or predictable forces on the pushing member 1706. The drive system1704 may increase accuracy of the instrument 1722. In some embodiments,rather than a user attempting fine control by different forces squeezingon a handle, the drive system may respond to operation of the userinterface in fine increments. For example, each operation of the userinterface 1702 may cause a partial actuation of the instrument 1722(e.g., advancing a movable portion of a Kerrison 1 mm for eachinteraction with the user interface 1702).

The fluid reservoir 1708 is fluidly coupled to the first or masterinflatable element 1710. The fluid reservoir 1708 may comprise a mass offluid such as saline, deionized water, etc., for example, contained inan intravenous bag positioned higher than the first inflatable element1710 such that gravity causes fluid to flow from the fluid reservoir1708 into the first inflatable element 1710, or may comprise apressurized canister and the like. After inflating or priming the firstinflatable element 1710, the fluid reservoir 1708 may be disconnectedfrom the first inflatable element 1710 (e.g., physically disconnectedfrom the first inflatable element 1710 or fluidly disconnected byclosing a valve between the fluid reservoir 1708 and the firstinflatable element 1710). The first inflatable element 1710 includes adefined volume to contain an amount of fluid in an inflated state. Insome embodiments, the first inflatable element 1710 includes aninflatable balloon similar to those used for percutaneous transluminalangioplasty or kyphoplasty. Certain such balloons are generallyinexpensive, disposable, sterile, and/or are not susceptible tooverinflation. The pusher member 1706 and the first inflatable element1710 may be at least partially contained by a housing 1712. The housing1712 may include apertures for connection to the fluid reservoir 1708and the fluid conduit 1714, and for the shaft of the pusher member 1706and/or to allow air and/or fluid to enter and exit the chamber as theair and/or fluid is displaced by inflation or deflation of the firstinflatable element 1710. The fluid conduit 1714 is in fluidcommunication with the first inflatable element 1710 and the second orslave inflatable element 1716.

As illustrated in FIG. 47B, with the first inflatable element 1710 atleast partially inflated by a fluid such as saline, for example afterbeing primed by the fluid reservoir 1708, the user interface 1702 isactuated, which causes the drive system 1704 to move the pusher member1706 to compress the first inflatable element 1710 within the housing1712, for example against a platen. Although illustrated as compressingthe first inflatable element 1710 radially inwardly (e.g., widthwise),longitudinal compression is also possible. Although illustrated ascompressing the first inflatable element 1710 in a single direction, aplurality of plungers or other compression mechanisms are also possible.For example, drive system 1704 may cause longitudinal compression of thefirst inflatable element 1710 from each end. For another example, drivesystem 1704 may cause inflation of a toroidal element around the firstinflatable element 1710. Fluid flows out of the first inflatable element1710 through the fluid conduit 1714 and into the second inflatableelement 1716. As the fluid enters the second inflatable element 1716,the second inflatable element 1716 inflates within the chamber 1718. Thechamber 1718 may restrict inflation of the second inflatable element1716 to be substantially linear or substantially in a single direction(e.g., towards the piston 1720). During inflation of the secondinflatable element 1716, the second inflatable element 1716 pushes thepiston 1720. The piston 1720 is coupled to the instrument 1722 or mayeven be part of the instrument 1722. As illustrated in FIGS. 47A and17B, the instrument 1722 is coupled to the chamber 1718. In someembodiments, the instrument 1722 is spaced from and/or separate from thechamber 1718. The chamber 1718 may include apertures for connection tothe fluid conduit 1714, and for the shaft of the piston 1720 and/or toallow air and/or fluid to enter and exit the chamber as the air and/orfluid is displaced by inflation or deflation of the second inflatableelement 1716.

The instrument 1722 may comprise any instrument actuatable by motion ofthe piston 1720 caused by the second inflatable element 1716. Forexample, the instrument 1722 may comprise a Kerrison, an aneurysm clipapplier, a rongeur, a tissue cutter, scissors, forceps, and othersurgical instruments. The instrument 1722 may be a non-surgicalinstrument, for example used for machinery, plumbing, electrical, andthe like. As described herein, hydraulic power can inhibit or eliminateinterference with electromagnetic tracking, so the instrument 1722 maycomprise any instrument used in a setting in which reduction ofinterference with electrical devices may be advantageous. In embodimentsin which the instrument 1722 comprises a Kerrison, the Kerrison mayinclude a D-shaped cutting surface, for example to cut bone without atwisting motion. In some embodiments, the cut bone fragments canaccumulate in a Kerrison lumen proximal to the cutting surface (e.g.,for later extrusion for removal such as by a screw auger), and the like.

The second inflatable element 1716, the chamber 1718, and the piston1720 may provide certain advantages over other systems. For example, ifthe system 1700 did not include the second inflatable element 1716 suchthat fluid flowed from the fluid conduit 1714 directly into the chamber1718, the chamber 1718, including interaction between the chamber 1718and the piston 1720, would need to be fluid-tight, but such systems areprone to leakage, especially at high pressure. A fluid-tight piston 1720may also cause stiction issues, leading to the use of higher pressure,which can disadvantageously lead to leakage. Lubricants that may reducethese disadvantages are generally not biocompatible. Stiction may alsolead to difficulty in precise movement of the piston 1720. By contrast,a system 1700 comprising a second inflatable element 1716 can allow thepiston 1720 to generally fit in the chamber 1718, but does not require afluid-tight fit. This can reduce or eliminate issues with stiction.Because the fluid is contained within the second inflatable element1716, issues with leakage of the chamber 1718 may be reduced oreliminated. Distal mechanisms such as diaphragms and bellows may alsohave issues. For example, a diaphragm generally has less range of motionthan an inflatable element 1716. For another example, bellows aregenerally expensive and therefore are possibly not disposable. Bycontrast, inflatable elements 1716 such as balloons may have a highrange of motion and/or be readily disposable.

In some embodiments, the second inflatable element 1716 comprises anexpandable elastomer (e.g., comprising flexible polyvinyl chloride(PVC), cross-linked polyethylene, polyurethane, polyethyleneterephthalate (PET), nylon, and/or other polymers). Nylon may be weakerand less compliant than PET, but may be softer and still thin and strongcompared to other materials. Advantages provided by PET over othermaterials can include tensile strength and/or maximum pressure rating.In some embodiments, the second inflatable element 1716 includes aninflatable balloon such as those used for percutaneous transluminalangioplasty or kyphoplasty. In some embodiments, the second inflatableelement 1716 is the same or substantially the same as or includes atleast one property (e.g., volume, radius, and/or length in an inflatedstate, type of material, etc.) as the first inflatable element 1710. Inembodiments in which the second inflatable element 1716 is the same orsubstantially the same as the first inflatable element 1710, actions onthe first inflatable element 1710 by the drive system 1704 may cause anequal but opposite effect on the second inflatable element 1716. Thismay be useful, for example, to visualize on the first inflatable element1710 what is happening to a non-viewable second inflatable element 1716.The use of a first inflatable element 1710 can provide a defined volume,for example to inhibit or prevent overinflation of the second inflatableelement 1716. In some embodiments, at least one of the first inflatableelement 1710, the second inflatable element 1716, the housing 1712, andthe chamber 1718 comprises a lubricious coating, for example to reducefriction between the first inflatable element 1710 and the housing 1712and/or between the second inflatable element 1716 and the chamber 1718.In some embodiments, at least one of the first inflatable element 1710and the second inflatable element 1716 comprises an abrasion andpuncture-resistant coating, for example to increase reliability.

Although illustrated in FIGS. 47A and 47B as having both ends as squareends, the either or both of the ends of the first inflatable element1710 and/or the second inflatable element 1716 may be any appropriateshape including, for example, conical sharp corner, conical radiuscorner, spherical end, and/or offset neck. For example, the firstinflatable member 1710 may include two offset necks (e.g., as in offsetor directional balloons) such that the connections to the fluidreservoir 1708 and the fluid conduit 1714 do not move during inflationand deflation of the first inflatable element 1710. Although illustratedin FIGS. 47A and 47B as having a uniform longitudinal profile, the firstinflatable element 1710 and/or the second inflatable element 1716 mayinclude longitudinal tapers, steps, combinations thereof, and the like.Although the first inflatable element 1710 is illustrated in FIGS. 47Aand 47B as inflating and deflating only radially and the secondinflatable element 1716 is illustrated in FIGS. 47A and 47B as inflatingand deflating in all directions, the first inflatable element 1710and/or the second inflatable element 1716 may be designed to inflateand/or deflate in a single dimension or direction (e.g., a singlelongitudinal direction, radially, etc.) or in all dimensions.

In some embodiments, the system 1700 optionally does not include thefirst inflatable element 1710. For example, upon receiving a signal fromthe user interface 1702, the drive system 1704 may cause fluid to flowinto the second inflatable element 1716 by operating a valve, advancinga piston within a cylinder (e.g., advancing the pusher member 1706within a fluid-tight housing 1712 in fluid communication with the fluidconduit 1710), or otherwise causing fluid to flow through the fluidconduit 1714.

In some embodiments, a biasing element such as a metal spring or aresilient elastomeric member can be positioned between the pusher member1706 and the housing 1712 and/or between the piston 1720 and the chamber1718. Such biasing element(s) can cause the system 1700 to be in adefault state when a user in not interacting with the user interface1702. For example, the pusher member 1706 can be biased away from thefirst inflatable element 1710 (e.g., the pusher member 1706 beingbetween a negative biasing element and the first inflatable element1710), and the force of the biasing element on the pusher member 1706provides available volume for at least partial inflation of the firstinflatable element 1716. For another example, the piston 1720 can bebiased towards the second inflatable element 1716 (e.g., the piston 1720being between a positive biasing element and the second inflatableelement 1716), and if the force of the biasing element on the piston1720 is sufficient to cause at least partial deflation of the secondinflatable element 1716, the second inflatable element 1716 can bebiased to a deflated state and the instrument 1722 can be biased to afirst state or open state in which inflation of the second inflatableelement 1716 is not providing force to the piston 1720.

In some embodiments, as illustrated in FIG. 47C, a spring member 1724(which can be a metal spring member, an elastomeric spring member, orany other suitable axially resilient member) can be positioned betweenthe piston 1720 and the chamber 1718 to cause the piston 1720 to returnto the default or initial position of the piston. In this arrangement,when the piston 1720 is in the default position (i.e., where the springmember 1724 is substantially fully expanded), a cutting head 1728 of theKerrison attached to the shaft 1726 will be in an open or retractedposition so as to be spaced apart from the fixed cutting surface 1730 ofthe Kerrison. As the second inflatable element 1716 is expanded, thepiston 1720 will be forced toward a first end 1720 a of the housing 1718against the bias of the spring member 1724, causing the cutting head1728 of the Kerrison to move toward the fixed surface 1730 of theKerrison, to effect the cutting of the bone or other tissue to be cutwith the Kerrison.

Additionally, with reference to FIGS. 47D-47G, the Kerrison can have avariety of cutting head configurations. For example, with reference toFIGS. 47E-47G, in any of the embodiments disclosed herein, the cuttinghead 1728 can have a circular shaped cross-section (as in FIG. 47E), thecutting head 1728 can have a C shaped cross-section (as in FIG. 47F) ora closed D shaped cross-section (as in FIG. 47G). Other shapes anddesigns are possible.

Additionally, in any of the Kerrison embodiments described herein, thefixed cutting surface 1730′ can be generally vertically oriented. Asshown in FIG. 47D, the fixed cutting surface 1730 can also be angulated(for example, angled at approximately 45 degrees, or from approximately35 degrees or less to 55 degrees or more).

In any of the tool embodiments disclosed herein, including withoutlimitation the Kerrison, the housing supporting or comprising the toolcan be configured to have a port or lumen therein arranged to facilitatethe removal of tissue and bone extracted from the surgical site. Forexample, the Kerrison can have a side port or opening located proximalof the cutting head 1728, 1728′, though which cut tissue can be removed(e.g., pushed through port or opening as cutter withdraws and Kerrisonreturns to the default position). In some embodiments, a source ofsuction, or a source of saline and suction, can be supplied to the port.Additionally, the removal port or lumen of the housing can also supporta mechanical removal mechanism, such as but not limited to a screw typeauger (which can be hydraulically actuated, via for example a gearmotor, gerotor, or vane motor 1512 discussed above), to facilitateremoval of bone debris and extracted tissue from the surgical site. Insome embodiments, the removed tissue can be extracted to a wastereservoir supported by or tethered to the housing of the tool. Inanother embodiment, the movable cutting head of the Kerrison can be agenerally cylindrical tube that can be actuated (in the matter describedabove) to slidably move against the fixed cutting surface 1730. Forexample, said cylindrical tube can be slidable within an outer housingof the Kerrison when a force is exerted thereon via the expansion of thesecond inflatable element 1716, as discussed above.

Additionally, in any of the tool embodiments disclosed herein, includingwithout limitation the Kerrison, the housing supporting or comprisingthe tool can be configured to have a suction port and a source of salineso that the tool and/or the surgical site can be flushed with saline andthe saline and debris can removed via the suction line simultaneously orsequentially with the flushing. In some embodiments, the saline can beprovided through the conduit used to provide saline to the secondinflatable element, through the same or a different lumen of suchconduit.

Additionally, the saline source or conduit and/or the suction source orconduit can be separate from the tool so that it can be independentlypositioned. In some embodiments, the saline source or conduit and/or thesuction source or conduit can be tethered to the tool.

Any of the hydraulic system embodiments disclosed herein can beconfigured to incorporate or use any suitable surgical tools, includingwithout limitation scissors, micro-scissors, forceps, micro-forceps,bipolar forceps, clip appliers including aneurysm clip appliers,ronguers, and, as described, Kerrison tools.

For example, FIGS. 48A and 48B show an embodiment of a hydraulicscissors tool 1732 that can be used with any of the hydraulic systemembodiments disclosed herein, including system 1700. The scissors toolcan have other configurations than shown in FIGS. 48A and 48B, and canbut is not required to have an adapted version of or some similarcomponents as a commercially available non-hydraulically operatedscissors tool.

In use, the second inflatable element 1716 can be actuated, as describedabove, to move the hydraulic micro-scissor tool 1732 from a first, openposition, as shown in FIG. 48A, to a second, closed position, as shownin FIG. 48B. In particular, in some embodiments, the second inflatableelement 1716 can be inflated to advance the piston 1720 toward thesecond end of the housing 1718. The piston can have one or more channels1721 formed therein. Each blade 1733 of the scissors can have a pin,dowel, tab, or other protrusion 1734 supported at an end thereof, thepins 1734 each being configured to translate along the length of thechannels 1721. In this arrangement, as the piston 1720 is advancedtoward the scissors 1732, the pins 1734 will move along the length ofthe channels 1721 and move inwardly, causing the blades 1733 of thescissors 1732 to come together, as illustrated in FIG. 48B. The scissorscan be in a fixed axial position relative to the piston 1720 to effectthe relative motion between the piston 1720 and scissors 1732 thatoperates the scissors. A spring or other biasing mechanism (for example,between the end of the housing and the piston) can be used to return thepiston back to the retracted position to return the scissors to an openarrangement or state.

FIGS. 49A and 49B show another embodiment of a hydraulic scissors tool1732′ that can be used with any of the hydraulic system embodimentsdisclosed herein, including system 1700. The scissors tool can haveother configurations than shown in FIGS. 49A and 49B, and can, but arenot required to, have an adapted version of or some similar componentsas a commercially available non-hydraulically operated scissors tool.With reference to FIGS. 49A and 49B a first scissors blade 1733 a can befixed or otherwise attached to the housing 1718, and the second scissorsblade 1733 b can be caused to move relative to the first scissors blade1733 a by moving the piston 1720 relative to the end of the housing,thereby causing a pin or other protrusion 1734 in the end of the handle1733 b to translate within the channel 1721, as described above withreference to FIGS. 48A and 48B. A spring or other biasing mechanism (forexample, between the end of the housing and the piston) can be used toreturn the piston back to the retracted position to return the scissorsto an open arrangement or state.

FIGS. 50A and 50B show another embodiment of a hydraulic scissors tool1732″ that can be used with any of the hydraulic system embodimentsdisclosed herein, including system 1700. The scissors tool can haveother configurations than shown in FIGS. 50A and 50B, and can, but isnot required, to have an adapted version of or some similar componentsas a commercially available non-hydraulically operated scissors tool.With reference to FIGS. 50A and 50B, a first scissors blade 1733 a canbe fixed or otherwise attached to the housing 1718, and the secondscissors blade 1733 b can be caused to move relative to the firstscissors blade 1733 a by moving the piston 1720 and shaft 1726 relativeto the end of the housing. The end of the shaft 1726 can have a slanted,curved, or angulated shape or surface that, when translated relative tothe second blade 1733 b, can cause the second blade 1733 b to rotaterelative to the first blade 1733 a, causing the cutting action. As withthe other embodiments disclosed above, in some embodiments, a springmember or other biasing mechanism 1724 can be positioned between the endof the housing 1718 and the piston 1720 to bias the piston 1720 to thefirst, retracted position, thereby biasing the scissors to an openposition.

Any other desired tool, such as a micro-scissors, forceps,micro-forceps, bipolar forceps, clip appliers including aneurysm clipappliers, ronguers, and, as described, Kerrison tools, can be configuredto work with any embodiments of the hydraulic system disclosed herein.Further as described above, any of the tools and configurations thatprovide actuation may be configured differently.

Additionally, the chamber or housing 1718 and shaft 1726 can beconfigured and adapted to work with and/or interchangeably support anysuitable tool, including without limitation scissors, micro-scissors,forceps, micro-forceps, bipolar forceps, clip appliers includinganeurysm clip appliers, ronguers, and, as described, Kerrison tools. Forexample, with reference to FIGS. 51A-51B, the system 1700 can beconfigured such that any of the aforementioned tools, or other suitabletools (represented schematically by the box 1740 can be removablyattached to an end portion 1718 a of the chamber 1718. In someembodiments, the tool 1740 can have an attachment element 1742 that canbe removably coupled with an end portion 1718 a of the chamber 1718. Anyof the tools used in this configuration can be modified from theirstandard form, as necessary, to be actuated with the linear moving shaft1726 or with a configuration similar to any other embodiment disclosedherein, such as without limitation the embodiments shown in FIGS.48A-50B.

In some embodiments, the attachment element 1742 for each of the toolscan be threadedly secured, or otherwise removably coupled, to the endportion 1718 a of the chamber 1718. However, the system 1700 can beconfigured such that the tool is integrally formed or otherwisenon-removably secured to the chamber 1718. Any desired tools, includingwithout limitation scissors, micro-scissors, forceps, micro-forceps,bipolar forceps, clip appliers including aneurysm clip appliers,ronguers, and, as described, Kerrison tools, can be similarly adapted tobe removably or non-removably coupled with the chamber 1718. Though insome embodiments, the surgical devices or tools can be modularly coupledto the hydraulic actuation system, in other embodiments the surgicaldevices or tools can at least partially incorporate one or morecomponents (e.g., chamber 1718, second inflatable element 1716), so thatthe tool and such components are part of a single piece.

The chamber 1718 can be contoured and shaped to fit comfortably withinthe hand of a surgeon, so that the surgeon can accurately control thepositioning and orientation of the tool. This chamber or housing 1718 isnot required to be cylindrical, but can have any suitable shape. Forexample, in some embodiments, the housing 1718 can have an ovularcross-sectional shape or any anatomically or aesthetically desirableshape. Additionally, the size of the housing 1718 can be varieddepending on the size requirements of the second expansion element 1716,the amount of force required to be exerted on the tool by the secondexpansion element and piston 1720, and other factors such as gripabilityand comfort for the user of the device.

Further to the description above, the operation of the tool attached tothe housing 1718 can be controlled by the positioning and orientation ofthe tool and by operation of the user interface, such as the userinterface 1702 discussed above. In some embodiments, the user interfacecan define a variety of different positions to enable the surgeon toaccurately control the amount of extension of the shaft 1726 and, hence,the state of the tool. Additionally, a spring mechanism or member can bepositioned within the chamber of any tool disclosed herein to bias thepiston 1720 and shaft 1726 toward a first or retracted position. In thisconfiguration, when the linear actuator is moved toward the relaxedstate, by operation of the user interface and the drive system 1704, thespring mechanism can cause the piston to compress the second inflatableelement of any tool disclosed herein to force the fluid within thesecond inflatable element through the conduit 1714 and back into thefirst inflatable element.

Some embodiments of the system 1700 can have a manifold 1760 in fluidcommunication with the conduit 1714, the manifold being configured todivert the fluid supplied through the conduit 1714 through any desirednumber of sub conduits. For example, with reference to FIG. 52, six ormore sub-conduits 1764 can be in coupled with the manifold 1760. Each ofthe different sub-conduits 1764 can be used to provide fluid to anydesired number of tools connected to the sub-conduits, each sub-conduit1764 being configured to couple with one tool. For example, a firstsub-conduit 1764 can be coupled with a Kerrison, a second sub-conduit1764 can be coupled with a rongeur, a third sub-conduit 1764 can becoupled with micro-scissors, a fourth sub-conduit 1764 can be coupledwith micro-forceps, and so on. In some embodiments, a first pneumaticport is directed toward powering scissors and forceps, a first hydraulicport is used to power a drill, and a second hydraulic port is used topower a Kerrison, with individual tubing for each flow path. In somesuch embodiments, individual tools can be replaced without replacingother tools or tubing associated with other tools.

Each of the variety of tools connected to the conduits can have asecondary inflation element, a chamber, a piston, and any othercomponents used to independently operate such tool, similar to thatdescribed in the embodiments above.

Additionally, though not required, each of the different sub-conduits1764 can have a flow valve 1762 associated therewith to independentlycontrol the flow of fluid through such sub-conduits 1764. A userinterface can be used to independently control the operation of theplurality of flow valves 1762 (e.g., via a controller, such as acomputer controller) such that a surgeon or other user canelectronically control the flow of fluid through the sub-conduits 1764using a control panel or other user interface. In this configuration,the surgeon or user can control the flow of fluid into and out of thesub-conduits 1764 and, hence, into and out of the second inflationelements 1716, of the respective tools, to exchange fluid between theprimary or first inflation element 1710 of the system and the secondinflatable elements 1716.

For example, in use, a user may wish to perform an operation with aKerrison coupled with a first sub-conduit 1764, but not a ronguer (e.g.,a second Kerrison) coupled with a second sub-conduit 1764. Even thoughboth such tools may be attached to the system, the user can essentiallyrestrict the flow of fluid from the first inflatable element to theronguer by closing the valve associated with the sub-conduit connectedto the ronguer. The valve associated with the first sub-conduit can beopened such that any fluid advanced into the manifold from the firstinflation element can fill the second inflation element of the Kerrisonto operate the Kerrison.

In some embodiments, the system can be coupled with a source of salineat the hospital or other facility to prime or prefill the firstinflatable element. Alternatively, the system can be prefilled withsaline such that the first inflatable element has enough saline or otherdesired fluid therein to completely fill all of the second inflatableelements in communication therewith. Though several embodiments ofhydraulically actuated surgical devices or tools are described above,one of skill in the art will recognize that the hydraulic actuationsystems described herein (e.g., master-slave balloon hydraulic system,vane motor hydraulic system) can be used to actuate a variety ofsurgical tools by using a hydraulically generated force or torque toturn, pivot, or otherwise move one mechanical component relative toanother to perform a desired surgical procedure (e.g., cut, hold,press).

In some embodiments, a hydraulically powered or actuated drill can beadapted to be used with any of the pressurized hydraulic systemsdisclosed herein, including without limitation any of the saline systemspre-charged by a bag compressor and/or roller pump, or otherwise, asdiscussed above. With reference to FIG. 53, some embodiments of thedrill 1850 can have a housing 1852, a drill chuck or other suitableconnector 1854 for receiving and supporting a drill bit or other desiredrotary bit 1856 therein, a turbine 1860 rotationally coupled with thechuck 1854. A first inlet fluid flow path or conduit 1862 connectable toa first or inlet conduit 1864 can direct a fluid, such as saline, towardthe turbine 1860 with a level of fluid flow velocity sufficient torotate the turbine and, hence the chuck and rotary bit 1856 at thedesired velocity. In some embodiments, the drill can have a hydrostaticbearing coupled with the turbine. In some embodiments, the drill canhave an air bearing coupled with the turbine. The air bearing and/orhydrostatic bearing can function as a thrust bearing. In someembodiments, the turbine used for the drill or any desired rotationaltool can be arranged similarly as compared to the impeller 2076 andnozzle frame 2072 described above, which can reduce the diameter or sizeof the turbine or impeller and the rotational tool. In this case, theoutput shaft 2079 can be coupled with a drill chuck or with a drill bitor other rotational tool.

A controller attached to a flow valve in the inlet line or conduit 1862or 1864 can be used to adjust the velocity of the flow against theturbine 1860. In some embodiments, a valve and valve controller (notillustrated) can be supported by or within the housing 1852 such that asurgeon can quickly and easily change the velocity of the drill byadjusting the valve in communication with the inlet flow conduit 1862. Aflow meter can be used to measure a flow velocity within the fluid flowpath.

In some embodiments, the outlet channel and conduit 1866, 1868 can havethe same cross-sectional size as the inlet conduit and channel,respectively. In some embodiments, the outlet channel and conduit 1866,1868 can have a larger cross-sectional size as compared to the inletconduit and channel, respectively, to reduce pressure buildup in theoutflow portion of the flow pathway.

The inlet and outlet lines can each have a one-way flow valve and quickconnector thereon to permit the drill to be removed from the hydrauliclines or system without the substantial loss of hydraulic fluid.Couplings 1870, 1872 can be positioned at the rear end of the housing1852 to removably couple the inlet and outlet conduits with the housing1852.

A saline or hydraulically powered tool, such as any of the toolsdescribed herein, can have several advantages. First, they can beconfigured to work with EM tracking without interference to the EMtracking, which electric powered drills other tools will not.Accordingly, the surgical tool, such as a disposable or non-disposablehydraulic drill or Kerrison could include or have attached thereto(e.g., via clip-on connector) an electromagnetic tracker sensor.Accordingly, the surgical tool, such as a disposable or non-disposablehydraulic drill or Kerrison could include or have attached thereto(e.g., via clip-on connector) an optical sensor such as a CCD or CMOScamera sensor or detector array. Additionally, such saline orhydraulically powered tools are lightweight, generate very little or noheat, use hydrostatic bearings, and produce little noise during use.

The hydraulic system may also be used for other purpose such as cleaningoptics and/or cooling light emitters for illuminating a surgical site.

As illustrated in FIG. 53A, a powered drill 1850′ (e.g., a hydraulicallyor electrically powered drill) can include a camera 1854 a. The camera1854 a can be positioned on the chuck 1854′ and/or on the handle 1857 ofthe drill 1850′. The camera 1854 a can be fixed in position rotationally(e.g., with respect to the axis of the rotary bit 1856′. The handle 1857can include a waist portion to inhibit axial movement of the hand of asurgeon or other practitioner during use of the drill 1850′. In someembodiments the drill 1850′ include a tactile feature 1857 a (e.g., afaceted ring, a nub, a protrusion, a roughened portion) fixed in arotational position on the handle 1857. The tactile feature 1857 a canprovide the practitioner with a tactile indication of the orientation ofthe camera 1854 a.

In some embodiments, as illustrated in FIG. 53B, a set of powered (e.g.,hydraulically, pneumatically, and/or electrically powered) scissors 1950comprises handles 1958 and a body 1957. A blade coupler 1954 can becoupled with the body 1957 and can provide mechanical coupling betweenthe body 1957 and a set of blades 1956. In some embodiments, the poweredscissors 1950 include a camera 1954 a mounted in a fixed rotationalposition (e.g., with respect to an axis of the blade coupler 1954). Insome embodiments, the coupler 1554′ can be rotated (e.g. using a gnarledring fixed thereto) to rotate the orientation of the blades 1956 withrespect to the handles 1958. In various embodiments, the orientation ofthe camera with respect to the handles 1958 remains intact. The poweredscissors 1950 can also include a tactile feature 1957 configured toprovide the user of the scissors with tactile confirmation of thealignment of the camera 1954 a. This tactile feature may, include but isnot limited to a facet, dimple or other surface.

A wide range of embodiments are therefore possible. Various embodimentsmay comprise, for example, a retractor having plurality of cameras thatform images and an image processing module and display configured suchthat the user selects which images formed by the cameras are displayed.These images may be video streams. In some embodiments, the user mayenlarge one or more of the images and/or display the one or more imagesmore centrally. For example, the user may select icons that representthe images and enlarge and place the image more central or at a desiredlocation. These icons may actually show the images, which as statedabove may comprise a video stream. The images or video stream may bereal time.

Various embodiments may comprise, for example, a retractor havingplurality of cameras having respective field-of-views and that areconfigured to obtain images as well as an image processing module anddisplay configured to display the images tiled in a geometricarrangement that is consistent with the field-of-views and/or positionsof the cameras. As described above, these images may be video streams.The geometric arrangement may include position and/or orientation of theimages.

Various embodiments may comprise, for example, a retractor and pluralityof cameras, wherein the retractor is configured to hold open a surgicalsite and the plurality of cameras are inwardly facing toward thesurgical site. The retractor may, for example, comprise a plurality ofblades having the cameras disposed thereon wherein the blades hold backtissue to hold open the surgical site. In some embodiments, the camerasface downward and inward into the surgical site. In some embodiments,the retractor includes proximal and distal cameras.

Various embodiments may comprise, for example, a surgical device such asa retractor and plurality of cameras comprising at least one distalcamera and at least one proximal camera. In some embodiments, theplurality of cameras comprise a plurality of distal cameras and/or aplurality of proximal cameras. In some embodiments, the proximal camerascan be disposed more toward the end of a retractor blade and the distalcameras can be disposed more toward the distal end of the retractor. Insome embodiments, the cameras face inward and possible downward into thesurgical site held open by the retractor.

Various embodiments may comprise, for example, a surgical devices suchas a retractor, a plurality of cameras, and a tracking system, whereinthe plurality of cameras comprises a first and a second camera and thetracking system is configured to track a first movement of the firstcamera and a second movement of the second cameral wherein the movementsare different. For example, the movements may be in different directionsand/or to different extents.

Various embodiments may comprise, for example, a surgical device such asa retractor having a plurality of cameras that form images and an imageprocessing module and display configured to display at least one of theimages and a tool rendered as partially transparent.

Various embodiments may comprise, for example, a surgical device such asretractor having a plurality of cameras that form images, and an imageprocessing module and a display configured to display an image morphedfrom at least two of the images.

Various embodiments may comprise, for example, a surgical devices suchas retractor having a plurality of cameras that form images, and animage processing module and a display configured to display at least twoof the images, wherein the at least two images are from cameras disposedon opposite sides of the surgical device (e.g. retractor) and whereinthe imaging processing module flips one of the two images.

Various embodiments may comprise, for example, a surgical device havinga plurality of cameras that form images, and a binocular display with atleast one internal display device coupled to the cameras to displayimages from the cameras. In some embodiments, the binocular display isdisposed on an articulated arm. In some embodiments, the binoculardisplay comprises a pair of oculars attached to a housing containing theinternal display device. In some embodiments, the binocular displaycomprises a non-direct view microscope comprising at least one cameracoupled to the binocular display (e.g., disposed at the bottom of thehousing) to provide images of the region below the binocular display.The images of the region below the binocular display may be displayed onthe at least one internal display device. The non-direct view microscopemay further comprise a variable zoom to alter the work distance of thenon-direct view microscope. The variable zoom may comprise a two stagezoom providing one control for changing the work distance and onecontrol for changing the magnification. In some embodiments, virtualreality gesture recognition may be provided to receive input fromgestures made below the binocular display. In some embodiments, thecamera below the binocular display may be used for such gesturerecognition. In some embodiments, the camera further comprises anadditional sensor (e.g. below the housing) for providing the gesturerecognition.

Accordingly, various embodiments may comprise, for example, a surgicaldevice having a plurality of cameras that form images, and a displaywith at least one internal display device coupled to the cameras todisplay images from the cameras and at least one sensor underneath toprovide a virtual reality graphic user interface. In some embodimentsthe sensor comprises an optical detector array for imaging. In someembodiments, the sensor comprises a distance measuring device.

Various embodiments may comprise, for example, a retractor havingplurality of cameras that form images and an image processing module anddisplay configured to provide a main wide field of view image withplurality of narrow field of view tiled images superimposed thereon. Theuser may select which images formed by the cameras are displayed. Theseimages may be video streams. The images or video stream may be realtime.

Various embodiments may comprise, for example, a retractor havingplurality of cameras that form images and an image processing module anddisplay configured to provide a composite main wide field of view imageproduced from plurality of images (e.g., stitched or tiled) with animage (e.g., picture-in-picture image) superimposed thereon. The usermay select which images formed by the cameras are displayed. Theseimages may be video streams. The images or video stream may be realtime.

Various embodiments may comprise, for example, a retractor havingplurality of cameras that form images and an image processing module anddisplay configured to provide a main wide field of view image from theretractor camera with an image superimposed thereon from a cameradisposed on a surgical tool. The user may select which images formed bythe cameras are displayed. These images may be video streams. The imagesor video stream may be real time.

Various embodiments may comprise, for example, a retractor having aplurality of cameras that form images and an image processing module anddisplay configured to provide multiple images from cameras on theretractor as icons showing images that can be enlarged and/or arrangedby a user. These images may be video streams. The images or video streammay be real time.

Various embodiments may comprise, for example, a retractor having aplurality of cameras that form images, a surgical tool, and an imageprocessing module and display configured to display camera images,wherein movement of the tool triggers selection of different sets ofcamera images. For example, the movement may comprise movement of thetool to different locations, e.g., to different depths within thesurgical site. These images may be video streams. The images or videostream may be real time.

Various embodiments may comprise, for example, a retractor having aplurality of cameras and a surgical tool having at least one camera thatform images.

Various embodiments may comprise, for example, a multiple pump hydraulicsystem for driving surgical tools, wherein the multiple pumps providesubstantially constant pressure to the system. The tools may be used inconjunction with a surgical device such as a retractor having camerasand a possible tracking system for tracking the cameras and/or tool.Proportional valves may be used to portion out pressurized fluid fromthe multiple pumps to the hydraulic tools. The multiple pumps maycombine pneumatics and hydraulics. The multiple pumps may be included ina disposable cassette.

Various embodiments may comprise a retractor, use a retractor, or areconfigured to be used with a retractor and not an endoscope,laparoscope, or arthroscope. Similarly, many embodiment compriseretractors or use retractors or are configured to be used withretractors wherein cameras are disposed on the retractors, notendoscope, laparoscope, or arthroscope.

In various embodiments the retractor fits in an opening in a manner toprovide ample room for the surgeon to operate but does not provide a gasseal for pumping up a cavity as may a laparoscope. Similarly, in manyembodiments the retractor does not maintain alignment of the layers oftissue that are cut through to form the incision.

In various embodiments, the retractor is not employed as a fulcrum forsurgical tools.

In various embodiments the camera(s) can be positioned on top of theretractor near and above the body surface or on the retractor at a depthwithin the surgical site (e.g., at the far distal end of the retractorinto the deepest portion of the surgical site, or elsewhere on theretractor such as more proximal).

Many embodiments are employed for spine surgery, neurosurgery, headand/or neck surgery, and ear nose and throat surgery and manyembodiments involve the cutting and extraction of bone, for example,through the pathway provided by the retractor.

Various embodiments, however, may be used with devices other thanretractors.

Many other embodiments are possible, including numerous combinations ofthe above recited features.

CONCLUSION

Various modifications to the implementations described in thisdisclosure may be readily apparent to those skilled in the art, and thegeneric principles defined herein may be applied to otherimplementations without departing from the spirit or scope of thisdisclosure. Thus, the claims are not intended to be limited to theimplementations shown herein, but are to be accorded the widest scopeconsistent with this disclosure, the principles and the novel featuresdisclosed herein.

Certain features that are described in this specification in the contextof separate embodiments also can be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment also can be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations may be described as occurring in aparticular order, this should not be understood as requiring that suchoperations be performed in the particular order described or insequential order, or that all described operations be performed, toachieve desirable results. Further, other operations that are notdisclosed can be incorporated in the processes that are describedherein. For example, one or more additional operations can be performedbefore, after, simultaneously, or between any of the disclosedoperations. In certain circumstances, multitasking and parallelprocessing may be advantageous. Moreover, the separation of varioussystem components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single product or packagedinto multiple products. Additionally, other embodiments are within thescope of the following claims. In some cases, the actions recited in theclaims can be performed in a different order and still achieve desirableresults.

What is claimed is:
 1. A medical apparatus comprising: a surgicalretractor; at least one video camera comprising imaging optics and anoptical sensor, said at least one camera disposed on said surgicalretractor, wherein said imaging optics comprises wafer-scale optics. 2.The medical apparatus of claim 1, further comprising a stop forward saidimaging optics.
 3. The medical apparatus of claim 2, wherein said sensoris proximal said imaging optics with said imaging optics between saidstop and said sensor, and wherein said stop the most distal opticalelement of said camera.
 4. The medical apparatus of claim 1, whereinsaid stop is just prior to said wafer optics.
 5. The medical apparatusof claim 4, further comprising a cover plate, said stop disposed betweensaid cover plate and said wafer-scale optics, said cover platecomprising sapphire.
 6. The medical apparatus of claim 1, furthercomprising a stop within said imaging optics.
 7. The medical apparatusof claim 6, wherein said stop is disposed within said wafer-scaleoptics.
 8. The medical apparatus of claim 1, further comprising amovable optical element within said imaging optics.
 9. The medicalapparatus of claim 8, wherein said movable optical element is withinsaid wafer-scale optics.
 10. The medical apparatus of claim 1, whereinsaid wafer-scale optics includes a movable optical element configured tobe moved to adjust said imaging optics.
 11. The medical apparatus ofclaim 10, further comprising an actuator configured to move said amovable optical element to adjust said imaging optics.
 12. The medicalapparatus of claim 1, wherein said imaging optics comprisesnon-wafer-scale lens elements.
 13. The medical apparatus of claim 12,wherein said imaging optics comprises a negative power non-wafer-scalelens element.
 14. The medical apparatus of claim 13, wherein saidnegative power non-wafer-scale lens element is disposed forward anywafer-scale optics such that said wafer-scale optics is disposed in anoptical path between said negative power non-wafer-scale lens elementand said optical sensor.
 15. The medical apparatus of claim 1, whereinsaid imaging optics comprises a negative lens group, a stop, and apositive lens group arranged in an optical path forward of said opticalsensor such that said stop and said positive group are disposed betweensaid negative lens group and said optical sensor.
 16. The medicalapparatus of claim 15, wherein said stop is between said negative lensgroup and said positive lens group.
 17. The medical apparatus of claim15, wherein said imaging optics provides a field of view of at least 70°and up to 125°.
 18. The medical apparatus of claim 1, wherein saidimaging optics comprises non-wafer-scale optics, a stop, and wafer-scaleoptics.
 19. The medical apparatus of claim 18, wherein said non-waferscale optics comprises negative optical power and said wafer-scaleoptics comprise positive optical power.
 20. The medical apparatus ofclaim 1, wherein said imaging optics comprises a stack ofnon-wafer-scale optics lenses having negative power, a stop, andpositive lens group.
 21. The medical apparatus of claim 1, wherein saidimaging optics comprises a front stop, a positive lens, a negative lens,and a plurality of lenses having positive power disposed in an opticalpath such that the negative lens is between the positive lens and theplurality of lenses having positive power.
 22. The medical apparatus ofclaim 21, wherein said imaging optics provides a field of view betweenabout 50°-70°.
 23. The medical apparatus of claim 1, wherein saidimaging optics comprises a front stop and four wafer-scale opticslenses.
 24. The medical apparatus of claim 23, wherein said imagingoptics comprises no more than four wafer-scale optics lenses.
 25. Themedical apparatus of claim 24, wherein said imaging optics provides afield of view between about 50°-70°.
 26. The medical apparatus of claim1, wherein said wafer-scale optics comprises a stack of wafer scaleoptics elements having air gaps therebetween, wherein said air gaps arein fluid communication with each other.
 27. The medical apparatus ofclaim 1, wherein said wafer scale optics comprises fiducials on platesthat provide stress to counteract bowing.
 28. The medical apparatus ofclaim 27, wherein said fiducials are interlocking.
 29. The medicalapparatus of claim 1, wherein said imaging optics are configured to bedisposed laterally with respect to said optical sensor to provide camerapointing.
 30. The medical apparatus of claim 29, further comprisingactuators to move said imaging optics laterally with respect to saidoptical sensor to provide camera pointing.
 31. The medical apparatus ofclaim 29, further comprising actuators to move said optical sensorlaterally with respect to said imaging optics to provide camerapointing.
 32. A medical apparatus comprising: at least one video cameracomprising imaging optics and an optical sensor; and a platformconfigured to be disposed on a surgical retractor, said at least onevideo camera disposed on said platform, wherein said imaging opticscomprises wafer-scale optics.